Architecture One-Pager

Spot Capacity Orchestrator · Solution Architecture v1.0 · Compute Platform Architecture · 2026-09 · 32 views · open source, fully on-premises

Every allocation is made on the assumption that it will be taken back, and nothing that has to happen in the next 120 seconds depends on a service that might not be there.

Most private clouds are sized for the peak of their guaranteed tenants and sit well below it most of the time. That idle headroom can run batch pipelines, training, CI and stateless services at a fraction of the price, provided it can be taken back the moment a guaranteed tenant wants it. The cheap part is lending the capacity. The expensive part is everything that makes borrowed capacity trustworthy: knowing which workloads can survive losing a node and which cannot, draining inside a window measured in seconds, not losing hours of work, noticing when a pool is being reclaimed far more often than its price suggests, falling back to guaranteed capacity without turning a shortage into an unbounded bill, and proving afterwards that the discount was larger than everything it cost. Teams that attempt this one workload at a time build five slightly different drain scripts, infer interruption tolerance from luck, and report a gross discount that ignores the redone work and the fallback spend. This platform exists to make the reliability of interruptible work a declared contract rather than an accident of which pool it landed in, and to make the saving a net, attributed number.

Every workload admitted to reclaimable capacity declares an interruption contract in a reviewed Git repository: its drain budget, how it checkpoints, its restart semantics, the interruption rate it can tolerate, its guaranteed floor, its deadline and who may authorise fallback spend. Most owners pick one of five tiers. A compiler written in CUE and Go rejects contracts the platform cannot honour and publishes signed bundles, and Kyverno and a Slurm job_submit filter refuse any work that arrives without one. In each of three regions a partitioned Go control plane, running on guaranteed capacity, turns pending work from Kubernetes and Slurm into normalised capacity claims, ranks about 140 OpenStack pools every five minutes by forward capacity, observed interruption and rate, excludes misbehaving pools within a minute, and places each workload's floor on committed capacity and the rest across at least eight pools, with no pool above 15% and no correlation group above 30%. Instances boot already holding the drain-relevant part of their contract. When the private cloud's reclaimer issues a 120-second notice, a node agent built in Go acts within two seconds from what it already holds: it taints the node, evicts through the scheduler or, if the scheduler does not answer, drains locally, and verifies the drain rather than assuming it. Workloads checkpoint through a platform client into a Ceph RGW bucket encrypted with their own key; Spark, PyTorch and Slurm jobs get this from integrations built once. The control plane learns of the loss from the lifecycle event and replaces it elsewhere. When a workload's contract would be breached, a fallback arbiter restores its floor, spends within its declared authority, and pages a named approver beyond it. Every placement, drain, loss and fallback becomes an event with a cause in Kafka and ClickHouse, and the saving is reported per workload net of wasted work, fallback, headroom, checkpoint storage and the orchestrator's own footprint.

What it is, and what it is not

• Reliability declared per workload in a reviewed contract — not tolerance inferred from a workload that happened to survive last month's reclaims.
• A drain decided on the node from state it already holds — not a notice forwarded to a central service that decides what the node should do.
• Diversification measured across groups that reclaim together — not a spread across pool names that share a host aggregate.
• Fallback bounded by a named person's spend authority — not automatic on-demand capacity with no ceiling, discovered on next month's bill.
• A saving reported net of every cost of the discount — not the gap between two rate cards multiplied by hours.
• Fail static when the control plane is gone — not a recovery that reconciles a day of drift in one burst.

The decisions that are the architecture

1. Split the planes at the notice (ADR-09) — The control plane decides where capacity comes from; the node decides what happens when it goes away. No interruption is handled by a call to a central service.
2. The contract is the interface (ADR-01) — Placement, drain, checkpoint and fallback are functions of a declared, reviewed contract. Changing behaviour means changing a file, never tuning the platform.
3. The provider is held to a written contract (ADR-05) — On premises, notice, advisory, rate card and forward capacity are built by the private cloud team to a specification, with the no-notice case as the safe floor when any of them fails.
4. Diversify against known topology first (ADR-06) — Correlation groups start from host aggregates and halls, which cause reclamation on premises, and are refined by observed co-reclamation rather than trusted to pool names.
5. The agent owns the deadline, the scheduler the mechanics (ADR-14) — Eviction goes through Kubernetes and Slurm when they answer inside 800 ms, and around them when they do not.
6. Checkpoints belong to the workload (ADR-18) — Application checkpoints through a platform client, in a bucket the orchestrator cannot read, with Spark, PyTorch and Slurm integrations built once.
7. The floor is bought and it works (ADR-22) — Guaranteed floors run real work on committed capacity; fallback beyond them spends only inside a named authority.
8. Savings are net or they are not savings (ADR-26) — Useful work at guaranteed prices minus every cost of running on reclaimable capacity, each charged to the workload that caused it.

Why it holds up over time

The instance types, the private cloud release, the schedulers and the checkpointing frameworks will all change several times over the life of this platform, and it will be run by people who were not in the room for any of these decisions. The design keeps its promises in a contract format, in events and in a small number of invariants that are tested, and keeps every product behind a seam narrow enough to replace without asking a workload team to change anything.

• The contract outlives every mechanism. A drain budget, a restart semantic and a floor mean the same thing whether the notice comes from OpenStack, a Kubernetes-native harvester or a public cloud's spot service. The contract file format is versioned and owned by this platform, so a change of provider changes an adapter, not 220 declarations.
• The plane split is a law of physics, not a preference. A notice window is short and a control plane can be down. That stays true whatever the window is and whatever runs the control plane, so the boundary drawn at the node does not age.
• Pools are data, not code. A new hardware generation, a fifth hall or a GPU family is a row in the pool catalogue with attributes the eligibility filter already understands. Nothing in scoring or placement names a SKU.
• Scale grows partitions, not loops. A new region or scheduler cluster adds partitions with their own leaders and token budgets. No component's workload grows with the size of the whole estate, so the architecture that serves 5,000 nodes serves 50,000 without a redesign.
• The ledger is built from events with causes. Rate cards, charging models and finance rules will change. Because every lost minute, fallback and placement is kept as a costed event with a cause, a new accounting rule is a new query over the same history, not a new system.
• The provider is replaceable behind a contract. The five provider commitments (notice, advisory, rate card, forward capacity, reclaim API) are written down. The same orchestrator can run against a different private cloud, or against a public cloud's spot service, by implementing those five interfaces.
• The invariants are exercised every day. Drain within budget, fail static without a control plane, no termination outside owned projects: the daily drill and the quarterly game day test these against production, so they survive team turnover and the well-meaning refactor that would otherwise erode them.
• The eligible population is the metric. The platform recommends moving work in both directions, off reclaimable capacity when the saving does not justify the risk and onto it when a guaranteed workload behaves as if it could tolerate interruption. It is judged by how much of the estate can safely use it, which keeps it useful as the estate changes shape.

Non-functional targets

The requirement's targets, how this design meets each one, and the view where the mechanism is drawn. Where a figure is a planning estimate, it is the first number the proof phase replaces.

Scope

In scope

• Interruption contracts, tiers, feasibility checks and admission refusal in each scheduler
• The pool model, eligibility, correlation groups, ranking, exclusion and concentration caps
• Acquisition against normalised capacity, replacement on notice, quota tracking and provider rate limits
• The node agent: notice handling, cordon, drain through or around the scheduler, verification and overrun records
• The checkpoint client, framework integrations and per-workload checkpoint buckets
• Guaranteed floors, capped automatic fallback, escalation, staggering and return
• Lifecycle events, net savings accounting, recommendations, audit of capacity actions and production drills

Explicitly out of scope

• How much capacity a workload needs, which remains with the schedulers and their autoscaling
• Placing work onto nodes, which remains with Kubernetes and Slurm
• The reclaim decision, the notice relay and the rate card, which the private cloud team builds to the provider contract
• Checkpoint contents and workload correctness beyond declared restart semantics
• Any managed cloud service; everything runs on hardware the organisation owns

Prove it before production depends on it

Eight weeks, one region, one Kubernetes cluster and one Slurm cluster of 800 reclaimable nodes between them, and three volunteer workloads: a Spark pipeline, a PyTorch training job and a CI runner pool. Each item replaces an assumption in this record with a measurement.

1. Deliver per-instance notices through the NATS relay to 5,000 simulated subscribers and to 800 real agents at once, and record p99 from reclaimer to agent
2. Reclaim all 800 nodes in one wave with the API server under normal load, and time taint, eviction and verified drain per node; repeat with the API server made unreachable
3. Write 320 concurrent 1.4 GB checkpoints through the client into RGW with SSE-KMS and record p99 commit time and gateway saturation
4. Decommission Spark executors holding the largest shuffle in the estate and record migration time against the 110-second budget
5. Kill the control plane for 12 hours during normal reclaims and confirm drains, overrun counts and the absence of any mass termination or acquisition on recovery
6. Run the ranking and planner on a replay of three months of reclamations and compare wasted work and concentration with a price-first planner

• Pass: four weeks of daily 1% drills and at least one hall-scale wave with drain overruns under 0.5%, wasted work under 3% and no tier-1 shortfall over 15 minutes.
• Pass: the net saving on the three volunteer workloads, measured by the ledger, is above 50% after every cost.
• Fail and rethink: the relay cannot deliver inside a second at fleet scale, which moves notice delivery onto a per-hypervisor channel exposed to the guest, or RGW cannot absorb the burst, which changes tier-1 checkpointing from final-write to periodic plus requeue.

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 33 records across 9 areas, each with the alternatives that lost and what the choice costs.