01Split the planes at the notice
The control plane decides where capacity comes from; the node decides what happens when it goes away. No interruption is handled by a call to a central service.
ADR-0932 views, in reading order. Every view ships three ways: an HTML page, an SVG that re-opens in diagrams.net fully editable, and draw.io source.
A platform service that runs interruptible work on capacity the organisation's own private cloud can take back, and keeps that work running well enough to be trusted. It chooses which pools of reclaimable capacity each workload runs on, drains and checkpoints work inside a 120-second notice, replaces what was lost, falls back to guaranteed capacity when a workload's declared contract would otherwise be breached, and reports the saving net of everything it cost. It serves 220 workloads on 12,000 to 40,000 vCPU across three regions the organisation owns, built from open-source software: OpenStack as the capacity provider, a node agent and partitioned controllers built in Go, Kubernetes and Slurm as the schedulers that keep placing the work, Ceph for checkpoints, and Kafka, ClickHouse and PostgreSQL for state and evidence. Read the set in order. Acts 1 and 2 fix the boundary and the people it serves. Act 3 shows the two planes and the contract that drives both. Act 4 separates four classes of data by what losing each costs. Act 5 walks placement, a notice, four drain paths, a checkpoint, a hall-wide reclamation, a fallback, a control-plane outage and a pool's exclusion. Acts 6 and 7 cover deployment, change, measurement, money, authority and every named failure. One rule governs every page: the control plane decides where capacity comes from; the node decides what happens when it goes away.
Every allocation is made on the assumption that it will be taken back, and nothing that has to happen in the next 120 seconds depends on a service that might not be there.
Most private clouds are sized for the peak of their guaranteed tenants and sit well below it most of the time. That idle headroom can run batch pipelines, training, CI and stateless services at a fraction of the price, provided it can be taken back the moment a guaranteed tenant wants it. The cheap part is lending the capacity. The expensive part is everything that makes borrowed capacity trustworthy: knowing which workloads can survive losing a node and which cannot, draining inside a window measured in seconds, not losing hours of work, noticing when a pool is being reclaimed far more often than its price suggests, falling back to guaranteed capacity without turning a shortage into an unbounded bill, and proving afterwards that the discount was larger than everything it cost. Teams that attempt this one workload at a time build five slightly different drain scripts, infer interruption tolerance from luck, and report a gross discount that ignores the redone work and the fallback spend. This platform exists to make the reliability of interruptible work a declared contract rather than an accident of which pool it landed in, and to make the saving a net, attributed number.
Every workload admitted to reclaimable capacity declares an interruption contract in a reviewed Git repository: its drain budget, how it checkpoints, its restart semantics, the interruption rate it can tolerate, its guaranteed floor, its deadline and who may authorise fallback spend. Most owners pick one of five tiers. A compiler written in CUE and Go rejects contracts the platform cannot honour and publishes signed bundles, and Kyverno and a Slurm job_submit filter refuse any work that arrives without one. In each of three regions a partitioned Go control plane, running on guaranteed capacity, turns pending work from Kubernetes and Slurm into normalised capacity claims, ranks about 140 OpenStack pools every five minutes by forward capacity, observed interruption and rate, excludes misbehaving pools within a minute, and places each workload's floor on committed capacity and the rest across at least eight pools, with no pool above 15% and no correlation group above 30%. Instances boot already holding the drain-relevant part of their contract. When the private cloud's reclaimer issues a 120-second notice, a node agent built in Go acts within two seconds from what it already holds: it taints the node, evicts through the scheduler or, if the scheduler does not answer, drains locally, and verifies the drain rather than assuming it. Workloads checkpoint through a platform client into a Ceph RGW bucket encrypted with their own key; Spark, PyTorch and Slurm jobs get this from integrations built once. The control plane learns of the loss from the lifecycle event and replaces it elsewhere. When a workload's contract would be breached, a fallback arbiter restores its floor, spends within its declared authority, and pages a named approver beyond it. Every placement, drain, loss and fallback becomes an event with a cause in Kafka and ClickHouse, and the saving is reported per workload net of wasted work, fallback, headroom, checkpoint storage and the orchestrator's own footprint.
The control plane decides where capacity comes from; the node decides what happens when it goes away. No interruption is handled by a call to a central service.
ADR-09Placement, drain, checkpoint and fallback are functions of a declared, reviewed contract. Changing behaviour means changing a file, never tuning the platform.
ADR-01On premises, notice, advisory, rate card and forward capacity are built by the private cloud team to a specification, with the no-notice case as the safe floor when any of them fails.
ADR-05Correlation groups start from host aggregates and halls, which cause reclamation on premises, and are refined by observed co-reclamation rather than trusted to pool names.
ADR-06Eviction goes through Kubernetes and Slurm when they answer inside 800 ms, and around them when they do not.
ADR-14Application checkpoints through a platform client, in a bucket the orchestrator cannot read, with Spark, PyTorch and Slurm integrations built once.
ADR-18Guaranteed floors run real work on committed capacity; fallback beyond them spends only inside a named authority.
ADR-22Useful work at guaranteed prices minus every cost of running on reclaimable capacity, each charged to the workload that caused it.
ADR-26The instance types, the private cloud release, the schedulers and the checkpointing frameworks will all change several times over the life of this platform, and it will be run by people who were not in the room for any of these decisions. The design keeps its promises in a contract format, in events and in a small number of invariants that are tested, and keeps every product behind a seam narrow enough to replace without asking a workload team to change anything.
A drain budget, a restart semantic and a floor mean the same thing whether the notice comes from OpenStack, a Kubernetes-native harvester or a public cloud's spot service. The contract file format is versioned and owned by this platform, so a change of provider changes an adapter, not 220 declarations.
A notice window is short and a control plane can be down. That stays true whatever the window is and whatever runs the control plane, so the boundary drawn at the node does not age.
A new hardware generation, a fifth hall or a GPU family is a row in the pool catalogue with attributes the eligibility filter already understands. Nothing in scoring or placement names a SKU.
A new region or scheduler cluster adds partitions with their own leaders and token budgets. No component's workload grows with the size of the whole estate, so the architecture that serves 5,000 nodes serves 50,000 without a redesign.
Rate cards, charging models and finance rules will change. Because every lost minute, fallback and placement is kept as a costed event with a cause, a new accounting rule is a new query over the same history, not a new system.
The five provider commitments (notice, advisory, rate card, forward capacity, reclaim API) are written down. The same orchestrator can run against a different private cloud, or against a public cloud's spot service, by implementing those five interfaces.
Drain within budget, fail static without a control plane, no termination outside owned projects: the daily drill and the quarterly game day test these against production, so they survive team turnover and the well-meaning refactor that would otherwise erode them.
The platform recommends moving work in both directions, off reclaimable capacity when the saving does not justify the risk and onto it when a guaranteed workload behaves as if it could tolerate interruption. It is judged by how much of the estate can safely use it, which keeps it useful as the estate changes shape.
The requirement's targets, how this design meets each one, and the view where the mechanism is drawn. Where a figure is a planning estimate, it is the first number the proof phase replaces.
| Quality | Target | How it is met | View |
|---|---|---|---|
| Notice to drain signal | ≤ 2 s p99; ≥ 115 s left | Per-hall NATS relay; decision from a local signed bundle; scheduler API capped at 800 ms before the local path | 18 |
| Final checkpoint | ≤ 45 s p99 for 2 GB | Multipart writes to an in-region RGW sized for 20 GB/s; manifest written last | 20 |
| Replacement serving | ≤ 180 s cold; ≤ 20 s warm | Replacement requested from the notice event in parallel with drain; warm headroom per contract in Phase 2 | 21 |
| Placement decision | ≤ 10 s p99 | Incremental planner over changed claims only; normalised units; pre-computed ranking | 17 |
| Ranking and exclusion | ≤ 5 min refresh; exclusion ≤ 60 s | Batch scorer every 5 minutes; streaming exclusion watcher on lifecycle events | 16 |
| Control plane availability | ≥ 99.9% monthly | Three replicas over three halls, leader per partition, Patroni with a synchronous replica | 25 |
| Node-local handling | ≥ 99.99%; ≥ 12 h without control plane | systemd agent with no central dependency; bundles never expire; local drain fallback | 23 |
| Guaranteed floor | ≥ 30% of tier-1 target | Floor filled first from committed reservations, spread over ≥ 3 halls, rebuilt within 15 minutes of a hall loss | 22 |
| Discount share | ≥ 70% of eligible hours | Tiers T2–T5 carry low or no floor; move-on recommendations grow the eligible population | 28 |
| Net saving | ≥ 55% blended | 59% planned: €199k of €335k counterfactual, net of all costs | 28 |
| Wasted work | ≤ 3% of reclaimable hours | Final write on notice; 60-minute periodic insurance; requeue units bounded by rate × length ÷ 2 | 20 |
| Drain overruns | ≤ 0.5% of reclamations | Feasibility check at compile time; overruns counted and contracts flagged | 12 |
| Capacity shortfall | ≤ 0.1% of capacity-hours; tier 1 > 15 min is an incident | Floor on committed capacity; capped automatic fallback; shortfall measured apart from interruption | 22 |
| Orchestrator cost | ≤ 2% of realised saving | Shared estate Kafka, ClickHouse and Prometheus; 20 m CPU agent; 1.6% planned | 28 |
| State recovery | RPO ≤ 30 s, RTO ≤ 10 min; rebuild ≤ 5 min | Synchronous replica; provider tags as reconstruction key; read-only rebuild before acting | 13 |
| Retention | Events 400 d; rankings 90 d / 13 mo; cost 7 y; checkpoints 3 gen or 7 d | ClickHouse TTLs; monthly Object Lock export; RGW lifecycle rules per bucket | 13 |
Eight weeks, one region, one Kubernetes cluster and one Slurm cluster of 800 reclaimable nodes between them, and three volunteer workloads: a Spark pipeline, a PyTorch training job and a CI runner pool. Each item replaces an assumption in this record with a measurement.
| Risk | If it lands | Response |
|---|---|---|
| The provider components are late or weaker than specified | Without a timely notice, every reclamation is a no-notice termination | The design is safe in the no-notice case by construction; tier contracts fall back to periodic checkpoints and requeue, at a higher wasted-work cost that the ledger reports (ADR-05, ADR-17) |
| Workload owners over-declare tolerance | Breaches, redone work and fallback spend land on workloads that should not be on reclaimable capacity | Breach alarms from measured rates, feasibility checks at compile time, fallback and waste charged to the owner, and contract-priced chargeback in Phase 3 (ADR-01, ADR-27) |
| Correlation is worse than the declared groups | A reclaim takes more than 30% of a workload's reclaimable capacity at once | The floor bounds the damage; learned co-reclamation groups in Phase 2 tighten the caps; the quarterly game day tests a hall-scale wave (ADR-06) |
| The checkpoint burst saturates RGW | Final writes miss the window during the largest reclaims | Store sized for twice the planned burst and measured in the proof; write failure releases the lease rather than losing work silently (ADR-19, ADR-20) |
| Rate-card savings never reach a purchase order | The platform reports savings that capacity planning does not act on | The ledger's reclaimable hours are published as an input to guaranteed-capacity planning, and the saving is labelled as rate-card until procurement uses it (ADR-26) |
| Built components become a bespoke burden | A small team owns an agent, a controller, a compiler and three client libraries | Each is small and tested by the daily drill; everything stateful and every protocol is adopted rather than built (ADR-10, ADR-13) |
Thirty-three decisions make up this architecture. Everything else on the thirty-two views is convention, and convention needs no defence. Each record opens with the question that forced a decision, says what was chosen and how it is built on hardware the organisation owns, lists the options that lost and why, and states what the choice costs. It then names the conditions under which a different organisation should choose differently, and explains why the choice should still hold after the products, the team and the estate have changed. Read the one-pager first. Read a record when you want to argue with it.
Status of this document. This is a design, not a report on a running system. Rates, latencies, volumes and costs are the requirement's stated assumptions or planning estimates derived from them, and the proof phase replaces each with a measurement. The design relies on behaviours that must be confirmed on the exact releases and hardware chosen before production depends on them: a NATS notice relay delivering a per-instance message to 5,000 subscribers inside one second at p99, including during a hall-wide reclaim; the Kubernetes Eviction API and taint updates completing for 800 nodes inside the drain budget without API-server throttling; kubelet graceful node shutdown honouring a systemd inhibitor delay of 110 seconds on the node images in use; a Kubernetes ValidatingAdmissionPolicy restricting a node-bound service-account token to taints and evictions on its own node; Spark executor decommissioning migrating shuffle to an S3-compatible fallback store within 90 seconds for the largest executors in the estate; Ceph RGW absorbing about 10 GB/s of multipart checkpoint writes from 320 concurrent writers with SSE-KMS through OpenBao; OpenStack application credentials and project-scoped roles preventing any other identity from deleting instances in the orchestrator's projects; and Nova multi-create and quota enforcement behaving under a token budget of 10 mutating calls a second per region. The provider-side components (the reclaimer, notice relay, rate card and reservation feed) do not exist in upstream OpenStack; this design specifies them and the private cloud team builds them.
What a workload must declare before it runs on capacity that can be taken back, how that declaration is checked, and where work without one is refused.
ADR-01A declared interruption contract is the only input to placement, drain, checkpoint and fallback ADR-02Five tiers cover most workloads, with a reviewed bespoke contract for the rest ADR-03Contracts live in Git, are compiled and checked for feasibility, and ship as signed bundles ADR-04Placement without a contract is refused inside each scheduler's admission pathWhat the private cloud promises, what a pool is, how pools that reclaim together are recognised, and how they are ranked and chosen.
ADR-05The capacity provider is the organisation's OpenStack cloud, held to a written provider contract ADR-06A pool is SKU family × size × hall × purchase model, and diversification is measured across correlation groups ADR-07Pools are ranked on a five-minute clock and excluded on a sixty-second stream, and every ranking is kept with its inputs ADR-08Placement prefers capacity depth over price, fills the floor first, and records what the preference costWhere the interruption boundary is drawn, how the deliberative part of the system is built and partitioned, where its state lives, and how it is kept from becoming the outage.
ADR-09The control plane decides where capacity comes from; the node decides what happens when it goes away ADR-10The control plane is partitioned, level-triggered controllers built in Go on a guaranteed management cluster ADR-11PostgreSQL holds fleet state for speed; tags on the provider's instances hold it for truth ADR-12Actuation is bounded: regional token budgets, termination breakers and gradual recoveryWhat runs on every reclaimable node, who owns the drain, what the node remembers when nothing else answers, and what happens when no notice comes at all.
ADR-13The node agent is a static binary under systemd, baked into images and configured at launch ADR-14The agent owns the trigger and the deadline; the scheduler's own eviction does the work, with a local fallback ADR-15The node caches only interruption-critical fields, signed, and never lets them expire ADR-16Proactive drain on an advisory is enabled per contract, and its false positives are charged to that contract ADR-17Notices arrive by stream with a metadata poll behind them, and the no-notice case is covered by leases, not by the agentWho checkpoints, where checkpoints live and who can read them, how often to take them, and how a resumed unit avoids doing anything twice.
ADR-18Checkpoints are application-level through a platform client, with frameworks implemented once ADR-19Checkpoints live in a Ceph RGW bucket per workload, encrypted with the workload's key and unreadable by the platform ADR-20The periodic checkpoint interval is set as a loss budget, not a habit ADR-21Restart semantics are declared, effects are published only on completion, and poison units are quarantinedHow much guaranteed capacity is held, who may spend beyond it, and how fallback and return avoid storms and oscillation.
ADR-22The reliability floor is bought, runs real work, and is drawn from committed capacity first ADR-23Fallback restores the floor without asking, spends automatically within a declared cap, and escalates beyond it ADR-24Fallback is staggered by priority and return waits for sustained healthHow every capacity event is recorded, how the saving is calculated honestly, who sees it, and who is woken when something goes wrong.
ADR-25Lifecycle events ride the estate's Kafka and ClickHouse; capacity actions are audited separately ADR-26The saving is computed net against a guaranteed-rate counterfactual, with every cost attributed ADR-27Showback per workload now; chargeback priced by the declared contract later ADR-28Metrics on Prometheus and Thanos, with alarms routed to whoever can fix the causeWho holds the power to acquire, terminate and spend, what a compromised node or workload can reach, and what happens to a reclaimed node's credentials.
ADR-29Capacity authority is scoped by OpenStack project, credentials are short-lived, and nodes hold none ADR-30Floors, spend caps and fallback policy need a second, higher authority ADR-31A terminated instance's identity and credentials are revoked, not left to expireHow the platform changes without becoming the cause of the interruptions it manages, and how its promises are tested on an ordinary day.
ADR-32Releases ride the fleet's own churn: canary partitions for contracts, new launches for agents ADR-33Reclamation is exercised daily in production through the provider's own reclaim pathEvery capability on the views, the package or design element that provides it, a credible alternative, and the record that justifies the choice. Everything runs on hardware the organisation owns. The provider-side components are built by the private cloud team to the provider contract, and are listed so the dependency is visible.
| Capability | Choice | Origin | Credible alternative | Why this one | Record |
|---|---|---|---|---|---|
| Interruption contract | Versioned YAML contracts, five tiers | This design | Inferred tolerance; platform knobs | Behaviour is a reviewable declaration | ADR-01 |
| Contract store and review | Forgejo, mirrored to a second region | Open source | Database behind an API | History, review and RPO 0 for free | ADR-03 |
| Contract compiler | CUE schemas plus Go feasibility rules | Open source | OPA / Rego | Arithmetic over pool history reads clearly | ADR-03 |
| Bundle signing | cosign with a key in OpenBao | Open source | Unsigned bundles over mTLS | Nodes verify what they act on | ADR-03 |
| Admission on Kubernetes | Kyverno validating policies | Open source | Built admission webhook | Refusal at submit time, fails closed | ADR-04 |
| Admission on Slurm | job_submit Lua plugin | Open source | Submission wrapper script | Cannot be bypassed by calling sbatch directly | ADR-04 |
| Capacity provider | OpenStack Nova, Placement, Ironic, Keystone | Open source | In-scheduler harvesting; public cloud spot | Existing regions, quotas and bare metal | ADR-05 |
| Reclaimer, rate card, advisories | Built by the private cloud team to contract | This design | Owned by the orchestrator | Lender decides reclaims, borrower reacts | ADR-05 |
| Committed capacity and forward calendar | OpenStack Blazar reservations | Open source | Static quota tables | Known future demand, already paid-for floor | ADR-22 |
| Notice delivery | NATS relay per hall, metadata poll behind it | Open source | Metadata polling; hypervisor channel | Sub-second push with a slower fallback | ADR-17 |
| Pool model and correlation groups | SKU × size × hall × model; aggregate and hall groups | This design | Independent pools | Diversify against known causes | ADR-06 |
| Scoring and exclusion | 5-minute scorer; streaming exclusion watcher | This design | Single fast loop | Judgement and reflex at their own speeds | ADR-07 |
| Portfolio planner | Depth-first water-filling with caps | This design | Price-first; global optimiser | Fewer interruptions, premium recorded | ADR-08 |
| Controllers | Go, partitioned, level-triggered, Kubernetes leases | This design | Temporal workflows; Karpenter fork | Rebuildable from the provider, scales by partition | ADR-10 |
| Management cluster | Kubernetes on guaranteed capacity, Argo CD | Open source | VMs under systemd | Standard operations for stateless controllers | ADR-10 |
| Fleet state and spend ledger | PostgreSQL 16 with Patroni and pgBackRest | Open source | etcd | Transactions, outbox and relational ledger | ADR-11 |
| Node agent | Static Go binary under systemd | This design | DaemonSet; kubelet extension | Works on every node type, without the scheduler | ADR-13 |
| Node images | diskimage-builder per SKU family | Open source | Packer | Native to OpenStack image pipelines | ADR-32 |
| Drain through Kubernetes | Eviction API; kubelet graceful node shutdown | Open source | Direct pod deletion | Native semantics with a local fallback | ADR-14 |
| Drain through Slurm | scontrol drain, requeueable jobs, USR1 | Open source | Kill and resubmit | Jobs resume rather than restart | ADR-14 |
| Checkpoint client | Go, Python and JVM libraries | This design | Per-team implementations | One manifest protocol and notice callback | ADR-18 |
| Framework checkpoints | Spark decommissioning, PyTorch Distributed Checkpoint | Open source | CRIU process snapshots | Small, fast and semantically correct | ADR-18 |
| Checkpoint storage | Ceph RGW, bucket per workload, SSE-KMS | Open source | CephFS; node-local NVMe | Durable, isolated, lifecycle rules per bucket | ADR-19 |
| Workload encryption keys | OpenBao transit, key per workload | Open source | One RGW-wide key | Orchestrator cannot read contents | ADR-19 |
| Floor and fallback | Floor keeper and fallback arbiter in the controller | This design | Manual fallback | Automatic within named authority | ADR-23 |
| Lifecycle events | Kafka (estate cluster), Vector on nodes | Open source | Direct writes to ClickHouse | Buffered, ordered per instance, shared cost | ADR-25 |
| History, ledger and reports | ClickHouse, Grafana | Open source | PostgreSQL for everything | Columnar history at 13-month and 7-year horizons | ADR-26 |
| Cost archive | Ceph RGW with Object Lock, monthly Parquet | Open source | ClickHouse only | Finance retention outside the query store | ADR-25 |
| Capacity action audit | Outbox to the estate audit log service | This design | Kafka telemetry | Independent of the telemetry pipeline | ADR-25 |
| Metrics and alerting | Prometheus, Thanos, Alertmanager | Open source | VictoriaMetrics | Estate standard, 13-month queries | ADR-28 |
| Workload and node identity | SPIRE, node-bound service-account tokens | Open source | Static node certificates | Revocable per node on termination | ADR-31 |
| Provider credentials | OpenStack application credentials via OpenBao | Open source | Long-lived service user password | One-hour, project-scoped, audited | ADR-29 |
| Human authentication | Estate OIDC provider with WebAuthn | Open source | Passwords | Phishing-resistant approval of spend | ADR-30 |
| Production drills | Exercise job calling the provider's reclaim API | This design | Chaos Mesh pod kills | Tests the real notice path, not a simulation | ADR-33 |
How does the platform know what a workload can tolerate, and what decides how it is placed, drained and protected?
How much of the contract should a workload owner have to compose themselves?
Where do contracts live, and how does the platform refuse a contract it cannot honour?
Where is work without a contract stopped from reaching reclaimable capacity?
On hardware the organisation owns, what plays the part of a cloud's spot market, and who builds the signals the requirement treats as provider mechanics?
What is the real unit of diversification, and can correlation between pools be known at all?
How quickly must the platform react to a pool going bad, and how can a bad placement be explained afterwards?
When a cheaper pool and a deeper pool are both eligible, which should the planner prefer, and how is that choice kept honest?
Should the response to a reclaim notice be decided centrally, with global knowledge, or locally, from what the node already holds?
How is the deliberative part of the system built, and how does it scale without becoming a single global loop?
Where does the orchestrator keep what it knows about the fleet, and what happens when that store is lost?
How is the orchestrator prevented from becoming the outage, through a request storm, a mass termination or a catch-up reconciliation after it returns?
What runs on each reclaimable node to handle notices, and how does it get its configuration before work arrives?
Who owns the drain: the orchestrator's node agent or the workload's existing scheduler?
Where does interruption state live so the fleet survives the orchestrator, and how stale may it be?
Should the platform drain on an early advisory, or only on the hard notice?
How does a notice reach a node inside a second, and what keeps work safe when no notice arrives at all?
Is checkpointing generic and platform-provided, or implemented by each application?
Where are checkpoints stored, who can read them, and how is storage kept from growing with the interruption rate?
How often should work be checkpointed when most reclaims come with notice?
How does a resumed or requeued unit avoid duplicating externally visible effects, and how does the platform stop paying for a unit that never finishes?
Is the reliability floor bought, or achieved by diversification?
Is fallback to guaranteed capacity automatic, budget-gated or human-approved?
How are fallback storms and oscillation between purchase models prevented?
How is every capacity event recorded durably at low cost, and why is the audit record kept apart from it?
How is the saving calculated so that it is true, and who pays for each cost of the discount?
Who holds the savings, and who holds the risk?
What is measured, and who is woken for which failures?
Who can launch and terminate reclaimable capacity, and how is a defect or compromise kept from reaching unrelated capacity?
Who may change the settings that determine how much the organisation spends on reliability?
What happens to the credentials a reclaimed node held?
How do contract changes and platform code reach thousands of nodes without the rollout becoming a source of interruptions?
How does the platform know its interruption path still works on a day when nothing is going wrong?
Every open-source package named on the views, what it is, the job it does in this design, and what was considered instead.
| Package | What it is | What it does here | Considered instead |
|---|---|---|---|
| OpenStack Nova | Compute service of the OpenStack cloud platform | Launches and deletes reclaimable, fallback and floor instances; multi-create for batched acquisition | Kubernetes-only harvesting |
| OpenStack Placement | Inventory and allocation service for OpenStack | Source of pool capacity and headroom per resource provider and aggregate | Static inventory |
| OpenStack Ironic | Bare-metal provisioning for OpenStack | Supplies bare-metal pools, including accelerator nodes | Metal3 |
| OpenStack Keystone | Identity service for OpenStack | Projects, roles and application credentials that scope capacity authority | Shared service user |
| OpenStack Blazar | Reservation service for OpenStack | Committed capacity for floors and the forward-capacity calendar | Quota tables |
| Kubernetes | Container orchestrator | Schedules services, Spark and CI onto reclaimable nodes; eviction and graceful node shutdown for drain; hosts the management cluster | Nomad |
| Slurm | Batch workload manager | Schedules batch and training jobs; drain, requeue and signals for interruption | HTCondor |
| Kyverno | Kubernetes policy engine | Refuses pods targeting reclaimable nodes without a contract | OPA Gatekeeper |
| Apache Spark | Distributed data-processing engine | Executor decommissioning migrates shuffle and cached blocks before a node is reclaimed | Restart failed stages |
| PyTorch Distributed Checkpoint | Sharded checkpointing for PyTorch training | Writes and restores training state through the checkpoint client | Framework-agnostic snapshots |
| CRIU | Checkpoint and restore of Linux processes | Phase 3 generic checkpoint for small CPU-only workloads that cannot be changed | Application checkpoints |
| Forgejo | Self-hosted Git forge | System of record for contracts, reviews and governance approvals | GitLab |
| CUE | Configuration and validation language | Contract and tier schemas in the compiler | JSON Schema |
| cosign | Signing tool from the Sigstore project | Signs contract bundles, with a second governance signature for money fields | GPG |
| NATS | Lightweight messaging system | Per-hall relay delivering notices and advisories to node agents | Instance metadata polling |
| PostgreSQL | Relational database | Fleet state, plans, exclusions, spend ledger and transactional outbox | etcd |
| Patroni | High-availability manager for PostgreSQL | Leader election and failover across three halls | Stolon |
| pgBackRest | Backup and restore for PostgreSQL | Point-in-time recovery of orchestrator state to Ceph | Barman |
| Apache Kafka | Distributed event log | Lifecycle event stream, ordered per instance | NATS JetStream |
| Vector | Observability data pipeline | Ships each node's event spool to Kafka when reachable | Fluent Bit |
| ClickHouse | Columnar analytical database | Event history, ranking history, cost attribution and savings ledger | PostgreSQL partitioned tables |
| Ceph RGW | S3-compatible object gateway for Ceph | Per-workload checkpoint buckets with SSE-KMS and lifecycle rules; Object Lock finance archive | MinIO |
| OpenBao | Secrets and encryption service, fork of Vault | Provider credentials, bundle signing keys, per-workload transit keys | HashiCorp Vault |
| SPIRE | SPIFFE workload identity runtime | Identities for controllers, agents and workloads; banned per node on termination | Static certificates |
| Prometheus and Thanos | Metrics system and its long-term store | Platform and workload metrics with 13-month queries | VictoriaMetrics |
| Alertmanager | Alert routing for Prometheus | Routes contract breaches to owners and platform faults to the SRE rota | Built alerting |
| Grafana | Dashboards and reporting | Workload savings reports, blast-radius board and fleet health | Superset |
| Argo CD | GitOps delivery for Kubernetes | Deploys controllers to regional management clusters by canary partition | Flux |
| diskimage-builder | OpenStack image building tool | Builds node images per SKU family with the agent baked in | Packer |
svg/<view>.svg or drawio/<view>.drawio in draw.io Desktop or at app.diagrams.net to edit. The SVG carries the diagram inside it, so it is both the picture and the source. This folder is self-contained — copy it whole and every link still resolves.