# Spot Capacity Orchestrator

**Solution Architecture v1.0 · open source, fully on-premises · Compute Platform Architecture · 2026-09 · 32 views · 33 architecture decision records**

A platform service that runs interruptible work on capacity the organisation's own private cloud can take back, and keeps that work running well enough to be trusted. It chooses which pools of reclaimable capacity each workload runs on, drains and checkpoints work inside a 120-second notice, replaces what was lost, falls back to guaranteed capacity when a workload's declared contract would otherwise be breached, and reports the saving net of everything it cost. It serves 220 workloads on 12,000 to 40,000 vCPU across three regions the organisation owns. It is built from open-source software: OpenStack (Nova, Placement, Ironic, Keystone, Blazar) as the capacity provider; a node agent and partitioned controllers built in Go; Kubernetes with Kyverno and Slurm as the schedulers that keep placing the work; Ceph RGW for checkpoints, with keys in OpenBao; PostgreSQL with Patroni for fleet state; Kafka, Vector and ClickHouse for lifecycle events and the savings ledger; and Forgejo with CUE for contracts.

The design rests on one rule: the control plane decides where capacity comes from; the node decides what happens when it goes away. No reclaim notice is ever routed through a central service before the node acts.

The decisions that carry the design:

- **The contract is the interface.** Placement, drain, checkpoint and fallback are functions of a declared, reviewed interruption contract. Tolerance is never inferred from a workload that happened to survive.
- **The provider is held to a written contract.** On premises, notice, advisory, rate card, forward capacity and a reclaim API are built by the private cloud team to a specification. When any of them fails, the design degrades to the no-notice case, which is safe by construction.
- **Diversify against known topology first.** Pools that share a host aggregate or a hall are reclaimed together, so no workload holds more than 30% of its reclaimable capacity in one such group. Learned co-reclamation refines the groups in Phase 2.
- **The agent owns the deadline; the scheduler does the eviction.** Kubernetes and Slurm drain the work when they answer within 800 ms, and the agent drains locally when they do not.
- **Checkpoints belong to the workload.** Application checkpoints go through a platform client into a bucket the orchestrator cannot read, with Spark, PyTorch and Slurm integrations built once.
- **The floor is bought, and it does work.** Tier-1 floors run real work on committed capacity. Fallback beyond the floor spends only inside a named person's authority.
- **Savings are net or they are not savings.** The saving is useful work valued at guaranteed prices, minus every cost of running on reclaimable capacity. Each cost is charged to the workload that caused it.

The architecture one-pager (including why the design holds up over time) and the full decision record appear on the landing page of the diagram set, directly below the index of views. The same content is published as [docs/architecture-one-pager.md](docs/architecture-one-pager.md) (~13 min) and [docs/decision-record.md](docs/decision-record.md) (~80 min).

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The landing page: 32 views in seven acts with every format linked, then the **architecture one-pager** and the **decision record** |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning cards, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same views as SVG with the diagram XML embedded; they re-open fully editable in diagrams.net |
| `diagrams/drawio/*.drawio` | draw.io source |
| `docs/architecture-one-pager.md` | The one-pager as markdown |
| `docs/decision-record.md` | The 33 decision records, the technology table and the package glossary as markdown |
| `specs/part-a..f.json` | Diagram specifications, the source of truth for every view, with every icon pinned |
| `specs/manifest-a..b.json` | Acts, page titles, subtitles and reasoning cards |
| `specs/adr-onepager.json`, `specs/adr-records-a..b.json` | The one-pager, the decision records, the capability-to-technology table and the glossary |
| `scripts/build.sh` | Rebuilds every deliverable from the specs (Node only) |
| `ask.md` | The requirement |

## Rebuild

```bash
bash scripts/build.sh
```

The build needs Node 20+ and the `architecture-diagram-portal` skill checked into this repository (override with `SK=<path>`). Every view passes the geometry check (0 errors, 0 warnings) and the routing check (211 edges, 0 errors, 0 warnings). Every icon resolves (492 nodes, 0 unresolved, 0 weak), and every relative link resolves.

## Notes on the requirement

- **Stack.** The requirement targets AWS: EC2 Spot, EKS node groups, a batch service and S3. As requested, this package runs on premises with open source. The spot market becomes reclaimable capacity on the organisation's OpenStack cloud, meaning headroom that guaranteed tenants have not claimed and that the private cloud takes back with 120 seconds' notice when they do (ADR-05). The EC2 interruption notice and rebalance recommendation become a per-hall NATS notice relay and an advisory, with instance metadata as a slower second channel (ADR-17). Spot prices become an internal rate card per pool. Capacity Reservations and Savings Plans become Blazar reservations (ADR-22). Capacity-optimised allocation becomes depth-first placement using the reservation calendar (ADR-08). EKS node groups become Kubernetes clusters on OpenStack. The batch service becomes Slurm. S3 checkpoints become per-workload Ceph RGW buckets with SSE-KMS through OpenBao (ADR-19). IAM tag conditions become dedicated OpenStack projects in which only the orchestrator holds a role (ADR-29).
- **The provider components do not exist upstream.** OpenStack has no preemptible-instance reclaimer, notice relay or pool rate card. This design specifies them as five commitments from the private cloud team (view 11, ADR-05) rather than pretending they are product features. If they arrive late, every reclaim is handled as a no-notice termination, with higher wasted work that the ledger reports.
- **The pool count does not reconcile with the stated dimensions.** Three regions × four zones × nine families × four sizes is 432 pools, not "roughly 140". The design reads 140 as roughly a third of family and size combinations being present in each hall, which is typical of an on-premises estate that holds three or four hardware generations per hall. Nothing depends on the exact figure; the planner's cost is per workload over 60 to 80 eligible pools.
- **A rate-card saving is not yet money.** On hardware the organisation owns, the 59% planned net saving (view 28) becomes cash only when capacity planning buys less guaranteed hardware as a result. The ledger publishes reclaimable hours to capacity planning and labels the figure as a rate-card saving (ADR-26).
- **Drills move into the MVP.** The requirement puts scheduled production reclamation exercises in Phase 2, but also asks that fail-static behaviour be verified by exercise in the MVP. A daily 1% drill through the provider's real reclaim path does both, at about 1% extra churn (ADR-33).
- **Advisory drain is not in the MVP.** Following the requirement's phasing, the contract compiler rejects drain budgets above 115 seconds until Phase 2. Spark workloads that would otherwise need longer get there through executor decommissioning (view 04).
- **Checkpoint interval.** Young–Daly, applied to the unprotected loss rate (no-notice, overrun and failed final writes, about 0.14% of instances an hour) rather than the 4% reclaim rate, gives an interval of about four hours. The default is capped at 60 minutes because the no-notice share is an assumption (ADR-20).
- **Hall loss.** The requirement asks that the guaranteed floor hold across the remaining zones. Holding a full floor through the loss of any hall would need a floor about 40% larger. The design instead spreads the floor so no hall holds more than 40% of it, and rebuilds the lost share from committed capacity within 15 minutes. It states that window as a risk (view 25, ADR-22).
- **Unproven behaviours.** Eight behaviours must be confirmed on the chosen releases and hardware before production depends on them. They include notice delivery at 5,000 subscribers, eviction of 800 nodes inside the budget, kubelet graceful shutdown with a 110-second inhibitor delay, and RGW absorbing about 10 GB/s of final checkpoints. They are listed in the decision record's status note, and each has a proof-phase item in the one-pager.