Architecture Decision Record

Spot Capacity Orchestrator · Solution Architecture v1.0 · Compute Platform Architecture · 2026-09 · 32 views · open source, fully on-premises

The argument these decisions serve is summarised in the Architecture One-Pager.

Thirty-three decisions make up this architecture. Everything else on the thirty-two views is convention, and convention needs no defence. Each record opens with the question that forced a decision, says what was chosen and how it is built on hardware the organisation owns, lists the options that lost and why, and states what the choice costs. It then names the conditions under which a different organisation should choose differently, and explains why the choice should still hold after the products, the team and the estate have changed. Read the one-pager first. Read a record when you want to argue with it.

Status of this document. This is a design, not a report on a running system. Rates, latencies, volumes and costs are the requirement's stated assumptions or planning estimates derived from them, and the proof phase replaces each with a measurement. The design relies on behaviours that must be confirmed on the exact releases and hardware chosen before production depends on them: a NATS notice relay delivering a per-instance message to 5,000 subscribers inside one second at p99, including during a hall-wide reclaim; the Kubernetes Eviction API and taint updates completing for 800 nodes inside the drain budget without API-server throttling; kubelet graceful node shutdown honouring a systemd inhibitor delay of 110 seconds on the node images in use; a Kubernetes ValidatingAdmissionPolicy restricting a node-bound service-account token to taints and evictions on its own node; Spark executor decommissioning migrating shuffle to an S3-compatible fallback store within 90 seconds for the largest executors in the estate; Ceph RGW absorbing about 10 GB/s of multipart checkpoint writes from 320 concurrent writers with SSE-KMS through OpenBao; OpenStack application credentials and project-scoped roles preventing any other identity from deleting instances in the orchestrator's projects; and Nova multi-create and quota enforcement behaving under a token budget of 10 mutating calls a second per region. The provider-side components (the reclaimer, notice relay, rate card and reservation feed) do not exist in upstream OpenStack; this design specifies them and the private cloud team builds them.

How to read a record

• Question: The forcing question: why a decision was needed at all.
• Context: The requirement, the scale and the constraint that make it hard.
• Decision: What this architecture does, stated so it can be checked.
• How it works on premises: The concrete mechanism: which package, configured how, on whose hardware.
• Options weighed: Chosen, rejected, deferred, or right elsewhere, with the reason for each.
• Consequences: What the choice buys and what it costs, both kept visible.
• Choose differently when: The conditions that would flip the decision for your system.
• Why it holds up over time: What keeps the decision right as scale, staff and technology change.
• Lesson: The principle that transfers beyond this platform.

Decision map

Contract and admission: What a workload must declare before it runs on capacity that can be taken back, how that declaration is checked, and where work without one is refused.

• ADR-01 · A declared interruption contract is the only input to placement, drain, checkpoint and fallback
• ADR-02 · Five tiers cover most workloads, with a reviewed bespoke contract for the rest
• ADR-03 · Contracts live in Git, are compiled and checked for feasibility, and ship as signed bundles
• ADR-04 · Placement without a contract is refused inside each scheduler's admission path

Provider and pools: What the private cloud promises, what a pool is, how pools that reclaim together are recognised, and how they are ranked and chosen.

• ADR-05 · The capacity provider is the organisation's OpenStack cloud, held to a written provider contract
• ADR-06 · A pool is SKU family × size × hall × purchase model, and diversification is measured across correlation groups
• ADR-07 · Pools are ranked on a five-minute clock and excluded on a sixty-second stream, and every ranking is kept with its inputs
• ADR-08 · Placement prefers capacity depth over price, fills the floor first, and records what the preference cost

Control plane: Where the interruption boundary is drawn, how the deliberative part of the system is built and partitioned, where its state lives, and how it is kept from becoming the outage.

• ADR-09 · The control plane decides where capacity comes from; the node decides what happens when it goes away
• ADR-10 · The control plane is partitioned, level-triggered controllers built in Go on a guaranteed management cluster
• ADR-11 · PostgreSQL holds fleet state for speed; tags on the provider's instances hold it for truth
• ADR-12 · Actuation is bounded: regional token budgets, termination breakers and gradual recovery

Node and drain: What runs on every reclaimable node, who owns the drain, what the node remembers when nothing else answers, and what happens when no notice comes at all.

• ADR-13 · The node agent is a static binary under systemd, baked into images and configured at launch
• ADR-14 · The agent owns the trigger and the deadline; the scheduler's own eviction does the work, with a local fallback
• ADR-15 · The node caches only interruption-critical fields, signed, and never lets them expire
• ADR-16 · Proactive drain on an advisory is enabled per contract, and its false positives are charged to that contract
• ADR-17 · Notices arrive by stream with a metadata poll behind them, and the no-notice case is covered by leases, not by the agent

Work preservation: Who checkpoints, where checkpoints live and who can read them, how often to take them, and how a resumed unit avoids doing anything twice.

• ADR-18 · Checkpoints are application-level through a platform client, with frameworks implemented once
• ADR-19 · Checkpoints live in a Ceph RGW bucket per workload, encrypted with the workload's key and unreadable by the platform
• ADR-20 · The periodic checkpoint interval is set as a loss budget, not a habit
• ADR-21 · Restart semantics are declared, effects are published only on completion, and poison units are quarantined

Floor and fallback: How much guaranteed capacity is held, who may spend beyond it, and how fallback and return avoid storms and oscillation.

• ADR-22 · The reliability floor is bought, runs real work, and is drawn from committed capacity first
• ADR-23 · Fallback restores the floor without asking, spends automatically within a declared cap, and escalates beyond it
• ADR-24 · Fallback is staggered by priority and return waits for sustained health

Evidence and money: How every capacity event is recorded, how the saving is calculated honestly, who sees it, and who is woken when something goes wrong.

• ADR-25 · Lifecycle events ride the estate's Kafka and ClickHouse; capacity actions are audited separately
• ADR-26 · The saving is computed net against a guaranteed-rate counterfactual, with every cost attributed
• ADR-27 · Showback per workload now; chargeback priced by the declared contract later
• ADR-28 · Metrics on Prometheus and Thanos, with alarms routed to whoever can fix the cause

Security and authority: Who holds the power to acquire, terminate and spend, what a compromised node or workload can reach, and what happens to a reclaimed node's credentials.

• ADR-29 · Capacity authority is scoped by OpenStack project, credentials are short-lived, and nodes hold none
• ADR-30 · Floors, spend caps and fallback policy need a second, higher authority
• ADR-31 · A terminated instance's identity and credentials are revoked, not left to expire

Operations: How the platform changes without becoming the cause of the interruptions it manages, and how its promises are tested on an ordinary day.

• ADR-32 · Releases ride the fleet's own churn: canary partitions for contracts, new launches for agents
• ADR-33 · Reclamation is exercised daily in production through the provider's own reclaim path

Technology by capability

Every capability on the views, the package or design element that provides it, a credible alternative, and the record that justifies the choice. Everything runs on hardware the organisation owns. The provider-side components are built by the private cloud team to the provider contract, and are listed so the dependency is visible.

The decisions, and the alternatives that lost

Contract and admission

What a workload must declare before it runs on capacity that can be taken back, how that declaration is checked, and where work without one is refused.

ADR-01 · A declared interruption contract is the only input to placement, drain, checkpoint and fallback

Status: Accepted · Shown on views: 04, 12, 14

How does the platform know what a workload can tolerate, and what decides how it is placed, drained and protected?

Context. Interruptible capacity is only useful if workloads that cannot survive interruption stay off it, and workloads that can survive it are handled in the way they need. The tempting shortcut is to infer tolerance: a job that has run on reclaimable capacity for a month without complaint is assumed to tolerate it. That confuses luck with design. A pool that happened to be quiet teaches nothing about the correlated reclaim that has not happened yet. The requirement asks for a declared contract, refused placement without one, re-evaluation against observed reality, and the contract as the single input to placement, fallback and drain.

Decision. Every workload on reclaimable capacity has a versioned contract stating drain budget, checkpoint mode and interval, restart semantics, maximum tolerable interruption rate, guaranteed floor, deadline if any, and the spend authority for fallback. Placement, eligible pools, drain grace, checkpoint trigger, fallback triggers and spend caps are computed from it and from nothing else. Observed behaviour is compared with the contract and raises a breach alarm; it never silently changes the contract.

How it works on premises. Contracts are YAML files in a Forgejo repository, one per workload, referencing a tier and overriding only permitted fields. The compiler resolves tiers and publishes a signed bundle per partition. The contract id and version travel as a Kubernetes label or a Slurm job comment, are stamped on every instance's tags and every lifecycle event.

What it buys

• Every behaviour can be explained by pointing at a line in a reviewed file
• Breach alarms compare measured reality with an explicit promise
• A change of provider or scheduler does not change what a workload declared

What it costs

• Onboarding is a pull request, not a checkbox
• Owners can declare optimistically, which needs alarms and eventually pricing to correct
• The contract schema becomes an interface that must be versioned carefully

Choose differently when. If nearly all interruptible work were one homogeneous kind, such as CI runners with short, idempotent jobs, a single platform policy with an opt-out would be simpler and nearly as safe.

Why it holds up over time. What a workload can tolerate is a property of the workload, not of the capacity it runs on. Whatever supplies the capacity in ten years, the same declaration still drives the same decisions, and the history of breaches against it remains meaningful.

Lesson. Surviving interruption once is an observation. Being designed to survive it is a declaration. Only the second can be relied on.

ADR-02 · Five tiers cover most workloads, with a reviewed bespoke contract for the rest

Status: Accepted · Shown on views: 04, 06, 12

How much of the contract should a workload owner have to compose themselves?

Context. A contract with seven fields and their interactions is too much to expect of 220 teams. The requirement asks for a small set of tiers so that most owners choose one, while permitting a bespoke contract where no tier fits. The tiers must be few enough to explain on one page and distinct enough that the choice matters.

Decision. Five tiers. T1 deadline: 30% floor, checkpointing required, automatic fallback within a spend cap. T2 checkpointed long-running: 10% floor, checkpointing required, fallback on breach within a cap. T3 requeue: no floor, no checkpoint, units bounded in length by the wasted-work budget. T4 stateless service: 30% floor, connection drain inside 60 s, replacement before loss. T5 opportunistic: no floor and no fallback, for CI and exploratory work that waits rather than spends. A bespoke contract starts from the nearest tier and is reviewed by the platform team.

How it works on premises. Tiers are CUE definitions in the contract repository. A workload file names a tier and may override only fields that tier allows; for example T3 may shorten its maximum unit length but cannot add a floor. Every tier change needs the same review as a bespoke contract.

What it buys

• Most onboarding is choosing a word
• Tier reports make fleet-level trade-offs visible
• Bespoke contracts are rare enough to review properly

What it costs

• Some workloads sit uneasily between tiers
• Tier definitions change rarely and need a migration when they do

Choose differently when. An estate with a very large number of teams and a strong platform-engineering culture might prefer composable profiles with policy checks instead of fixed tiers.

Why it holds up over time. The tiers describe kinds of work that have existed for decades: deadline batch, long computation, short idempotent units, stateless services and best-effort jobs. The numbers inside each tier will be tuned; the five kinds will not disappear.

Lesson. A choice between five named options gets made carefully. A form with seven numeric fields gets filled in with defaults.

ADR-03 · Contracts live in Git, are compiled and checked for feasibility, and ship as signed bundles

Status: Accepted · Shown on views: 07, 12, 26

Where do contracts live, and how does the platform refuse a contract it cannot honour?

Context. The requirement asks that a drain budget longer than the notice window be treated as a request for proactive draining, that a checkpoint which cannot be written inside the drain budget be rejected, and that the contract and policy store have an RPO of zero. A contract that is syntactically valid can still be impossible, and finding that out during a reclaim is the most expensive way to learn it.

Decision. Contracts are files in a Git repository and nowhere else. A compiler validates them against the schema and runs feasibility rules: drain budget above 115 s requires advisory drain; the declared final checkpoint must fit the budget at measured store throughput; a requeue unit must satisfy rate × length ÷ 2 ≤ wasted-work budget; at least eight eligible pools must exist. It replays 30 days of pool history to show expected placement and fallback frequency in the pull request, then emits a bundle per partition signed with cosign.

How it works on premises. Forgejo holds the repository, mirrored to a second region. Forgejo Actions runs the compiler, written as CUE schemas plus Go feasibility rules, on self-hosted runners. Bundles are signed with a key held in OpenBao and served by the contract distributor over HTTPS.

What it buys

• A rejected contract says exactly which rule failed and how to fix it
• Contract history is the audit trail of reliability decisions
• Nodes verify what they run

What it costs

• Feasibility rules must be kept in step with measured store and API performance
• Owners need Git access and some fluency with YAML

Choose differently when. If contracts changed many times a day under automation rather than by people, an API with the same compiler behind it would fit better than pull requests.

Why it holds up over time. Git, a schema and a signature are the most stable interfaces in software. The compiler's rules will evolve, but the pattern of reviewing a declaration and refusing what cannot be honoured does not depend on any product.

Lesson. The cheapest place to find out a promise cannot be kept is before anyone relies on it.

ADR-04 · Placement without a contract is refused inside each scheduler's admission path

Status: Accepted · Shown on views: 07, 10, 12

Where is work without a contract stopped from reaching reclaimable capacity?

Context. The platform does not place work; the schedulers do. If the only guard were in the orchestrator, any pod that tolerated the reclaimable taint or any job submitted to a reclaimable partition could land there directly, contract or not, and nobody would know until it was reclaimed.

Decision. Reclaimable nodes carry a taint in Kubernetes and belong to dedicated partitions in Slurm. Kubernetes admission refuses any pod that tolerates the taint without a contract label referencing a published contract version. Slurm's job_submit filter refuses jobs to reclaimable partitions without a contract comment. Both check against the current bundle, which they hold locally.

How it works on premises. Kyverno validating policies in every Kubernetes cluster, reading a ConfigMap synchronised from the bundle. A Lua job_submit plugin on each slurmctld, reading the same bundle from disk. Both fail closed for reclaimable targets and do not affect guaranteed workloads.

What it buys

• No path onto reclaimable capacity bypasses the contract
• Owners get the refusal at submit time, not at the first reclaim

What it costs

• Two admission implementations to keep in step
• A bundle sync failure blocks new reclaimable work, by design

Choose differently when. If a single scheduler ran all interruptible work and exposed a native quota or flavour mechanism tied to policy, admission could be expressed there instead of in a separate policy engine.

Why it holds up over time. Every scheduler that will ever run this work has an admission hook of some kind. The rule, no contract no placement, is small enough to port to whatever replaces Kyverno or Slurm.

Lesson. A rule enforced only by the system that is not in the path is a suggestion.

Provider and pools

What the private cloud promises, what a pool is, how pools that reclaim together are recognised, and how they are ranked and chosen.

ADR-05 · The capacity provider is the organisation's OpenStack cloud, held to a written provider contract

Status: Accepted · proof required · Shown on views: 01, 08, 11

On hardware the organisation owns, what plays the part of a cloud's spot market, and who builds the signals the requirement treats as provider mechanics?

Context. The requirement was written for EC2 Spot, where reclamation notices, rebalance recommendations, prices and capacity signals exist. On premises none of these exists until someone builds it. The private cloud already has regions, halls as availability zones, flavours, projects with quotas, host aggregates, bare metal through Ironic and reservations through Blazar. What it lacks is a way to lend unreserved headroom and take it back with notice when guaranteed tenants need it.

Decision. The private cloud team runs reclaimable capacity as a purchase model on its existing aggregates and commits, in writing, to five interfaces: a hard notice at least 120 s before deletion, delivered to the instance within one second through a per-hall NATS relay and mirrored in instance metadata; an advisory when guaranteed demand is forecast to reclaim a pool, when it can be forecast; a rate card per pool every five minutes; reservation and forward-capacity data from Blazar; and a reclaim API that the exercise job may call. The orchestrator consumes these and does not decide what to reclaim.

How it works on premises. OpenStack Nova, Placement, Ironic, Keystone and Blazar as the substrate. The reclaimer is a service owned by the private cloud team, following the pattern of CERN's preemptible-instance work, which selects victims when a guaranteed request cannot be placed and publishes notices. The guaranteed tenants' promise, capacity back inside three minutes, sets the 120-second window.

What it buys

• The signals are better than a public cloud's: aggregates and reservations are known facts
• The lender's and borrower's promises are written down separately
• The same orchestrator can run against another provider that meets the five interfaces

What it costs

• Five provider components have to be built by another team, on their roadmap
• Notice delivery at fleet scale is unproven until measured
• The 120-second window is a negotiated number, not a law

Choose differently when. If the estate ran one scheduler over all its hardware, harvesting inside that scheduler with priority preemption would remove the need for instance-level reclaim entirely.

Why it holds up over time. A borrower of reclaimable capacity always needs notice, a price and a view of future supply. Writing those as interfaces rather than as calls to a product means the orchestrator survives a change of private cloud, a move to a public cloud, or both at once.

Lesson. When the platform you depend on is run by colleagues, write the contract anyway. It is the only thing that survives a reorganisation.

ADR-06 · A pool is SKU family × size × hall × purchase model, and diversification is measured across correlation groups

Status: Accepted · Shown on views: 16, 21, 24

What is the real unit of diversification, and can correlation between pools be known at all?

Context. Pools are only useful for spreading risk if they fail independently. On a public cloud, correlation is unpublished and has to be learned slowly from events that may be too rare to learn from. On premises the cause of reclamation is known: guaranteed demand landing on a host aggregate, or a hall losing power or cooling. Pools that share an aggregate or a hall reclaim together by construction, whatever their flavour names say.

Decision. A pool is the narrowest unit reclaimed independently: server SKU family, flavour size, hall and purchase model. Pools are grouped into correlation groups by shared host aggregate and by hall from provider topology in the MVP. In Phase 2, observed co-reclamation (the phi coefficient of two pools being reclaimed in the same ten-minute window over 90 days) splits or merges groups. Each workload keeps at least eight pools, no more than 15% of its reclaimable capacity in one pool and no more than 30% in one group.

How it works on premises. The pool catalogue is synchronised from Nova flavours, Placement resource providers and aggregate membership. Group membership and phi values are stored with each ranking so a placement can be explained by the groups in force when it was made.

What it buys

• A reclaim of one aggregate is bounded at 30% of any workload's reclaimable share
• Correlation evidence is kept with every decision

What it costs

• Aggregate changes by the cloud team change groups and can force re-planning
• Caps sometimes exclude the cheapest capacity

Choose differently when. On a public cloud, where aggregate membership is invisible, observed co-reclamation and conservative spreading by instance family and zone become the only tools and should be the default from day one.

Why it holds up over time. Hardware generations, flavours and halls will change; the rule that two pools which fail together count as one does not. Groups are data refreshed from topology and events, so new hardware joins the model without code.

Lesson. Two pools that are reclaimed together are one pool, whatever the catalogue calls them.

ADR-07 · Pools are ranked on a five-minute clock and excluded on a sixty-second stream, and every ranking is kept with its inputs

Status: Accepted · Shown on views: 09, 16, 24

How quickly must the platform react to a pool going bad, and how can a bad placement be explained afterwards?

Context. The requirement asks for rankings refreshed at least every five minutes, exclusion within 60 seconds of a threshold, automatic return after a cool-off, and a versioned record of scoring inputs so that a bad decision can be traced to the signal that produced it. A single loop cannot do both jobs well: ranking needs history and is expensive; exclusion needs to be immediate and is simple.

Decision. A scorer runs every five minutes per region and writes a ranking with its inputs: rate card, 30-day interruption rates, forward capacity and quota headroom. An exclusion watcher consumes lifecycle events continuously, trips a pool on three reclamation waves in 30 minutes or on acquisition thrash, excludes it immediately, and applies a two-hour cool-off that doubles on repeat, then an hour of probation capped at 5% of any workload. Rankings are retained 90 days in full and 13 months aggregated.

How it works on premises. The scorer is a Kubernetes CronJob reading ClickHouse and the provider's feeds and writing to PostgreSQL and ClickHouse. The exclusion watcher is a Kafka consumer group per region with state in PostgreSQL. Both are part of the controller codebase in Go.

What it buys

• A pool going bad is out within a minute
• Any placement can be replayed with the data available at the time

What it costs

• Ranking history is the largest data set the platform keeps
• Thresholds need tuning against real reclamation patterns

Choose differently when. A small estate with a few pools and rare reclaims could rank hourly and exclude by hand without losing much.

Why it holds up over time. The split between slow judgement and fast reflex is independent of the signals used. Predictive scoring in Phase 3 replaces the scorer's formula and leaves the exclusion reflex and the record untouched.

Lesson. Keep the decision and the evidence together. A ranking without its inputs cannot be argued with, only obeyed.

ADR-08 · Placement prefers capacity depth over price, fills the floor first, and records what the preference cost

Status: Accepted · Shown on views: 16, 17, 14

When a cheaper pool and a deeper pool are both eligible, which should the planner prefer, and how is that choice kept honest?

Context. On premises a pool is cheap because guaranteed tenants are not using it yet; the forward calendar often shows they are about to. Choosing by price alone steers work towards the pools most likely to be reclaimed. The requirement asks for allocation strategies that favour depth, with the cost of that preference recorded so it can be reviewed.

Decision. The planner fills a workload's claim in a fixed order: guaranteed floor from committed reservations; then reclaimable units by composite score, where forward capacity and observed interruption outweigh rate; water-filled under the pool and group caps. Pools priced at or above 85% of the guaranteed rate are not treated as discount pools. For every plan the planner records the depth premium: the difference between what it chose and the cheapest eligible placement under the same caps.

How it works on premises. Implemented as a greedy water-filling allocation in the controller, which is fast enough for 60 to 80 eligible pools per workload in milliseconds. The depth premium is written to the plan row and summed monthly in the ledger.

What it buys

• Interruption rate and wasted work fall
• Reviewers can see whether depth is still worth what it costs

What it costs

• A measurable premium over the cheapest placement
• Per-workload greedy fills can be locally suboptimal

Choose differently when. Workloads that are nearly free to interrupt, such as T5 opportunistic jobs, can reasonably use price-first placement; the tier can say so.

Why it holds up over time. The trade between price and depth exists in every market for interruptible capacity. Recording the premium means the preference can be re-weighted from evidence as the market changes, instead of being argued from memory.

Lesson. A preference whose cost is not written down becomes a habit nobody can justify or remove.

Control plane

Where the interruption boundary is drawn, how the deliberative part of the system is built and partitioned, where its state lives, and how it is kept from becoming the outage.

ADR-09 · The control plane decides where capacity comes from; the node decides what happens when it goes away

Status: Accepted · Shown on views: 02, 07, 08, 18

Should the response to a reclaim notice be decided centrally, with global knowledge, or locally, from what the node already holds?

Context. A notice arrives, and every choice in response to it is better made with global knowledge: how long to allow, whether to checkpoint, where the replacement goes. So the tempting design reports the notice to the orchestrator, which decides and instructs the node. That sequence must complete inside 120 seconds for thousands of nodes at once during a correlated reclaim, quite possibly while the same event is degrading the control plane. It works perfectly in the steady state and fails exactly when it is needed, which is the failure this platform exists to prevent for its workloads.

Decision. Split the two planes on the interruption boundary. Placement, scoring, acquisition, replacement and fallback are global and deliberative, live in the control plane, and their unavailability costs convergence. Cordon, drain, checkpoint trigger, verification and lease release are local and immediate, and are executed by the node agent from a contract it received before any work landed. No reclaim notice is ever routed through a central service before the node acts. Replacement is requested from the lifecycle event, in parallel, and nothing on the node waits for it.

How it works on premises. The node agent receives notices from the provider's per-hall relay and holds a signed contract bundle on local disk. The control plane runs per region on a guaranteed management cluster and learns about each notice from Kafka. The two share no synchronous call on the notice path.

What it buys

• Fail static is a property of the architecture, not an aspiration
• Drain scales with the number of nodes, because each node does its own
• The control plane can be upgraded or lost without affecting a single drain

What it costs

• The node's decision uses cached, possibly stale, contract fields
• The drain cannot use global knowledge, such as a better-than-usual replacement being imminent

Choose differently when. If notice windows were hours rather than seconds and reclaims never correlated, a central decision with global knowledge would be both safe and better.

Why it holds up over time. The argument rests on two facts that do not age: a notice window is short, and any central service can be down. Every future change either honours the boundary, with the interruption path ending at the node, or contradicts it by drawing an arrow from a notice into a central service, which makes the rule easy to review against.

Lesson. Put each decision where it can still be made on the worst day, not where it would be made best on an average one.

ADR-10 · The control plane is partitioned, level-triggered controllers built in Go on a guaranteed management cluster

Status: Accepted · Shown on views: 09, 25

How is the deliberative part of the system built, and how does it scale without becoming a single global loop?

Context. The requirement asks for scaling by fleet partition rather than one global reconciliation loop, cost of evaluation proportional to change, bounded provider API use, and addition of a region or account without redesign. The provider's view of what exists is authoritative, and the state the orchestrator holds must be reconstructible from it.

Decision. One controller binary per region, with a leader per partition, where a partition is one scheduler cluster in one region. Each controller reconciles desired plan against observed instances and acts on the difference; there is no long-running workflow state. The controllers run on a management Kubernetes cluster on guaranteed capacity across three halls, never on the capacity they manage.

How it works on premises. Go, using controller-runtime-style reconciliation with Kubernetes Lease objects for per-partition leadership. The OpenStack Go SDK (gophercloud) for Nova, Ironic, Placement and Blazar. Deployed with Argo CD to the regional management cluster.

What it buys

• A stuck partition affects one scheduler cluster's capacity
• A controller restart loses nothing
• New regions and clusters are new partitions

What it costs

• A bespoke controller codebase to own
• Cross-partition decisions, such as contended floors, need explicit coordination later

Choose differently when. An estate with only Kubernetes and a single cluster might reasonably build on Karpenter's provisioning model with a custom OpenStack provider and a contract admission layer.

Why it holds up over time. Level-triggered reconciliation against an authoritative external state is the pattern that has outlived every orchestration fashion since Borg. Partitioning by scheduler cluster follows the estate's own seams, which change more slowly than any product.

Lesson. When someone else holds the truth, do not keep your own copy of the journey. Look again, compare, and act on the difference.

ADR-11 · PostgreSQL holds fleet state for speed; tags on the provider's instances hold it for truth

Status: Accepted · Shown on views: 13, 14, 23

Where does the orchestrator keep what it knows about the fleet, and what happens when that store is lost?

Context. The requirement asks for continuous reconciliation against the provider's authoritative state, reconstruction of all fleet state after a total loss of the orchestrator's store within five minutes and without operator input, an RPO of 30 seconds and an RTO of 10 minutes.

Decision. Plans, rankings in force, exclusions, the spend ledger and the event outbox live in PostgreSQL per region. Every instance the orchestrator launches carries tags naming workload, pool, partition and plan generation. On a total loss, a new database is rebuilt by listing instances in the orchestrator's projects and reading their tags; plans are recomputed from contracts and rankings. Termination requires the tags and the database row to agree.

How it works on premises. PostgreSQL 16 managed by Patroni across three halls, one synchronous and one asynchronous replica, pgBackRest to Ceph for point-in-time recovery. Instance tags through Nova server tags and metadata.

What it buys

• Losing the database loses speed and history, not ownership
• A mistaken termination needs two independent records to be wrong

What it costs

• Tags are part of the provider contract and must not be edited by others
• The spend ledger's recent entries depend on the synchronous replica

Choose differently when. If the provider could not tag instances reliably, the orchestrator would need its own durable inventory with a stricter RPO and a periodic diff against provider listings.

Why it holds up over time. Any capacity provider worth using can label what it runs. Using those labels as the key to rebuild state keeps the orchestrator's store a cache of convenience across any change of database.

Lesson. Store your own view for speed, but make sure the world can tell you what you own when that view is gone.

ADR-12 · Actuation is bounded: regional token budgets, termination breakers and gradual recovery

Status: Accepted · Shown on views: 17, 21, 23

How is the orchestrator prevented from becoming the outage, through a request storm, a mass termination or a catch-up reconciliation after it returns?

Context. The requirement asks that the orchestrator rate-limit its own acquisition, bound provider API calls under all conditions including recovery, fail static with no mass termination or acquisition, and recover gradually because a control plane that reconciles a large drift at once is indistinguishable from an outage of its own. The provider's API is shared with every guaranteed tenant, including the ones reclaiming capacity.

Decision. Each region has a token budget of 10 mutating provider calls a second, split statically between partitions, with Nova multi-create to launch many instances per call. Terminations are capped at 2% of a partition's fleet per 10 minutes; exceeding it halts termination and pages. After any restart the controller rebuilds state read-only for two cycles, then acts on at most 5% of a partition per 5 minutes, tier 1 first. An apparently orphaned instance is terminated only after its scheduler reports it idle on three observations.

How it works on premises. Token buckets in the controller keyed by region and partition; breaker state in PostgreSQL; limits agreed with the private cloud team and recorded in the provider contract.

What it buys

• The orchestrator's worst case is a known number of calls a second
• Recovery cannot become a second incident

What it costs

• Replacement during a very large reclaim is slower than the provider could serve
• Static splits leave budget idle in quiet partitions

Choose differently when. A provider with per-tenant API isolation and generous limits would allow dynamic budgets without risk to other tenants.

Why it holds up over time. Shared control-plane APIs are a commons in every cloud, public or private. Limits that are explicit, agreed and enforced by the caller remain correct whatever the provider's own throttling does.

Lesson. Automation that can act on the whole fleet needs a speed limit it enforces on itself.

Node and drain

What runs on every reclaimable node, who owns the drain, what the node remembers when nothing else answers, and what happens when no notice comes at all.

ADR-13 · The node agent is a static binary under systemd, baked into images and configured at launch

Status: Accepted · Shown on views: 10, 07, 17

What runs on each reclaimable node to handle notices, and how does it get its configuration before work arrives?

Context. The agent must work on Kubernetes nodes and on Slurm nodes, keep working when the scheduler it drains through is unavailable, and never be in a state where work is running and the drain budget is unknown. A DaemonSet is the usual way to run a node component on Kubernetes, but it depends on the kubelet and scheduler it may need to work around, and it does not exist on Slurm nodes.

Decision. A single static Go binary, running as a systemd unit, is baked into every reclaimable image. The acquirer passes the contract subset for the workloads the instance is intended for in cloud-init user-data, so the agent holds a valid drain configuration from boot. The agent then refreshes the signed bundle every 60 seconds. It has a budget of 20 millicores and 40 MiB.

How it works on premises. Images built with diskimage-builder in the release pipeline for each SKU family, one for Kubernetes nodes and one for Slurm nodes, with the same agent binary. Metrics on a local Prometheus endpoint; events to a bounded disk spool shipped by Vector.

What it buys

• The same drain logic serves every workload type
• No window between boot and a known drain budget

What it costs

• Agent upgrades need new images or in-place restarts
• Image builds become part of the platform's release

Choose differently when. An estate that ran only Kubernetes with immutable node images might run the agent as a static pod, gaining Kubernetes tooling while keeping independence from the scheduler.

Why it holds up over time. systemd and a statically linked binary will run on whatever operating system these nodes use for the next decade. Configuration at launch through user-data is supported by every provider, public or private.

Lesson. The component that must work when everything else is failing should depend on as little of everything else as possible.

ADR-14 · The agent owns the trigger and the deadline; the scheduler's own eviction does the work, with a local fallback

Status: Accepted · proof required · Shown on views: 18, 19

Who owns the drain: the orchestrator's node agent or the workload's existing scheduler?

Context. Kubernetes and Slurm already have cordoning, eviction and graceful termination. Draining through them reuses semantics workloads understand and prevents two systems evicting the same pod, but couples the notice path to a scheduler control plane that may be slow or unavailable inside 120 seconds at fleet scale. Draining purely from the node is fast and independent but races the scheduler, which may place new work on the node or read the disappearance as a failure.

Decision. The agent decides and keeps the deadline; the scheduler executes. On notice the agent marks the node unschedulable, then evicts or drains through the scheduler with a grace period equal to the contract's drain budget, then signals the workload. If the scheduler API does not answer within 800 ms, the agent drains locally: on Kubernetes through a systemd shutdown inhibitor and the kubelet's graceful node shutdown by pod priority, on Slurm by signalling the job's cgroup. A PodDisruptionBudget is honoured until the drain budget is spent. The same path serves reclaims, voluntary rebalances, operator drains and drills.

How it works on premises. Kubernetes: a node-bound service-account token with a ValidatingAdmissionPolicy allowing taint and eviction on its own node only; kubelet GracefulNodeShutdown configured with priority-based grace periods. Slurm: scontrol drain through slurmctld, requeueable jobs with a USR1 signal, local cgroup signalling as fallback. Spark: executor decommissioning with fallback storage on Ceph.

What it buys

• Workloads see the same termination semantics they already handle
• Drain completes with the scheduler unavailable
• One implementation for containers and batch jobs

What it costs

• Two drain paths to test, which the daily drill exercises
• In the local path the scheduler learns late and may briefly show the node as failed

Choose differently when. If the scheduler's eviction path could be shown to complete within the budget for thousands of nodes during an API-server brownout, the local fallback would be insurance rarely used and could be simplified.

Why it holds up over time. Any future scheduler will have its own eviction semantics and its own availability. Owning the deadline while delegating the mechanics keeps the agent's contract the same for whichever scheduler comes next, needing only a new adapter.

Lesson. Use the system that knows the work to stop the work, but never give it the power to miss the deadline.

ADR-15 · The node caches only interruption-critical fields, signed, and never lets them expire

Status: Accepted · Shown on views: 10, 12, 23

Where does interruption state live so the fleet survives the orchestrator, and how stale may it be?

Context. The agent has to act alone, so it holds a copy of contract fields that can be stale, and a stale copy is exactly what will be used during an outage. The options are a thin agent with a hard dependency, a fat agent with a full copy, or a middle position. The requirement also asks whether an agent with no control plane should drain more conservatively or exactly as it always does.

Decision. The node caches only the fields the notice path needs: drain budget and signal, checkpoint mode, restart semantics, advisory-drain enablement and scheduler adapter settings. Placement, eligibility, floors, spend and fallback fields are never cached, so a stale cache cannot affect them. The cache is a signed bundle with no expiry; staleness is reported as a metric, not enforced. With no control plane, the agent drains exactly as it always does.

How it works on premises. Bundle on local disk under /var/lib/sco, verified against the bundle-signing public key embedded in the image, refreshed every 60 seconds from the contract distributor. Initial copy from cloud-init user-data.

What it buys

• A 12-hour control-plane outage does not change a single drain
• Stale fields are the ones reviewed and changed least often

What it costs

• A shortened drain budget reaches old nodes up to 60 seconds late, or much later during an outage
• Bundle signing keys must be rotated with image rebuilds in mind

Choose differently when. If contracts changed drain behaviour often, for example daily, bounded staleness with a conservative default would be worth its complexity.

Why it holds up over time. Which fields a disconnected node needs is determined by the notice path, which is stable. Keeping the list short keeps the reasoning about staleness short, whatever the contract grows to include.

Lesson. Cache what you need to act, not everything you know. What is not cached cannot be stale.

ADR-16 · Proactive drain on an advisory is enabled per contract, and its false positives are charged to that contract

Status: Accepted · Phase 2 · Shown on views: 12, 29

Should the platform drain on an early advisory, or only on the hard notice?

Context. The advisory arrives a median of six minutes before a reclaim when it arrives at all, and it is absent for 35% of reclaims. Acting on it gives long-draining workloads the time they need, but the advisory is a probability: draining on it interrupts work that would have survived, and at a high false-positive rate the platform becomes the main source of interruption in its own fleet.

Decision. Hard notice only in the MVP; contracts with drain budgets above 115 seconds are rejected. In Phase 2, advisory drain is an opt-in contract field. The platform measures advisory precision per pool, drains opted-in workloads when an advisory arrives, and charges the compute interrupted by advisories that were not followed by a reclaim to that workload's ledger as a false-positive cost.

How it works on premises. Advisories arrive on the same NATS relay as notices, on a separate subject. Precision is computed in ClickHouse from advisory and reclaim events per pool over 30 days and published with each ranking.

What it buys

• Long-draining workloads become eligible
• The cost of a noisy signal lands on whoever chose to use it

What it costs

• More wasted work in pools with poor advisory precision
• The MVP excludes some workloads it could serve later

Choose differently when. If the provider's advisory proved highly precise, for example driven by reservation start times rather than forecasts, it could become the default for tiers 1 and 2.

Why it holds up over time. Every advisory signal from every provider will be probabilistic. Deciding per contract and pricing the false positives keeps the choice correct as the signal's quality changes.

Lesson. A probabilistic warning is worth acting on only for those who can pay for being wrong, and they should see that bill.

ADR-17 · Notices arrive by stream with a metadata poll behind them, and the no-notice case is covered by leases, not by the agent

Status: Accepted · proof required · Shown on views: 10, 18, 32

How does a notice reach a node inside a second, and what keeps work safe when no notice arrives at all?

Context. The drain budget leaves two seconds from notice to signal. OpenStack's instance metadata service caches responses for 15 seconds by default and polling it every second from 5,000 instances puts heavy load on the Nova API. The requirement also assumes 2% of reclaims arrive with no notice at all, and requires every guarantee to stay safe in that case.

Decision. The provider publishes notices on a per-hall NATS relay, on a subject per instance, to which the agent keeps a subscription. The agent also polls instance metadata every five seconds as a slower second channel. The no-notice case is not handled by the agent: work items are held on leases that expire, periodic checkpoints bound lost work, schedulers requeue units on node loss, and the lifecycle event records the termination as no-notice with its wasted work.

How it works on premises. NATS clusters per hall run by the private cloud team, with per-instance credentials issued at launch. Kubernetes Jobs use a pod failure policy that treats disruption as retryable; Slurm jobs are submitted requeueable; queue consumers use the queue's own visibility timeout.

What it buys

• Notice delivery does not depend on one channel
• The no-notice case uses mechanisms that already exist in every scheduler

What it costs

• The relay is a provider component that must be proven at fleet scale
• No-notice losses are only as small as the checkpoint interval makes them

Choose differently when. A provider with a local, push-style notice path per instance, as public clouds offer through their metadata services, would remove the need for a separate relay.

Why it holds up over time. Treating the no-notice case as the baseline and the notice as an optimisation means any failure of the delivery channel degrades to a known, measured state instead of an unhandled one.

Lesson. Design for the notice that never arrives. Then every notice that does arrive is a bonus, not a dependency.

Work preservation

Who checkpoints, where checkpoints live and who can read them, how often to take them, and how a resumed unit avoids doing anything twice.

ADR-18 · Checkpoints are application-level through a platform client, with frameworks implemented once

Status: Accepted · Shown on views: 19, 20, 07

Is checkpointing generic and platform-provided, or implemented by each application?

Context. A platform checkpoint, a process or memory snapshot taken without the workload's cooperation, onboards workloads that will never be modified, at the cost of large images, writes that may not fit the window, and restore fidelity that is hard to guarantee, especially with GPUs, open sockets and in-flight network state. An application checkpoint is small, fast and semantically correct, but asks each workload to implement it. Most interruptible work in this estate runs on a handful of frameworks the platform can reach: Spark, PyTorch and Slurm-managed jobs.

Decision. Checkpoints are application-level. The platform provides a client library that handles the notice callback, multipart upload, encryption, manifest commit and restore lookup, and integrates the frameworks once: Spark executor decommissioning with fallback storage, PyTorch Distributed Checkpoint with torchrun elastic restart, and Slurm requeue with a signal. Generic process checkpointing with CRIU is deferred to Phase 3, for workloads whose state is small and CPU-only.

How it works on premises. Client libraries in Go, Python and JVM published to the estate's package registries. Framework integrations shipped as base images and Helm chart defaults so a Spark or PyTorch workload gets them by using the platform's image.

What it buys

• Checkpoints fit the drain budget
• Restore correctness is the workload's own logic, testable by its owner
• Framework users need no code

What it costs

• Bespoke applications must call the client
• The platform owns three client libraries and three integrations

Choose differently when. If most interruptible work were unmodifiable third-party binaries with small in-memory state, generic snapshots would become the main path rather than the exception.

Why it holds up over time. Frameworks already expose checkpoint hooks because failure is normal in distributed computing. Building on those hooks, rather than on process internals, keeps checkpoints working across kernel, runtime and accelerator changes.

Lesson. The cheapest checkpoint is the one written by the code that knows which state matters.

ADR-19 · Checkpoints live in a Ceph RGW bucket per workload, encrypted with the workload's key and unreadable by the platform

Status: Accepted · proof required · Shown on views: 13, 20, 21

Where are checkpoints stored, who can read them, and how is storage kept from growing with the interruption rate?

Context. The requirement asks for checkpoints on durable storage independent of the instance, in the workload's own storage boundary, encrypted with its key, unreadable by the orchestrator, isolated so one workload's restore cannot read another's state, bounded to three generations or seven days, and never on instance-local storage unless declared disposable. The worst load is not steady state but a correlated reclaim, when hundreds of final writes land at once.

Decision. Each workload has its own RGW bucket, with a bucket policy granting access to the workload's identity only and SSE-KMS using a per-workload key. Generations are written as parts plus a manifest written last; a lifecycle rule keeps the newest three manifests and expires anything older than seven days, including orphaned parts. The orchestrator's identity has no read grant and sees usage statistics only. Each region's RGW is sized for 20 GB/s of checkpoint ingest, twice the planned burst of an 800-node reclaim.

How it works on premises. Ceph RGW in each region with SSE-KMS backed by OpenBao transit, workload access through STS AssumeRoleWithWebIdentity using the workload's SPIFFE JWT. Gateways on dedicated nodes with 100 GbE, placed in the same halls as the reclaimable pools they serve.

What it buys

• A compromised orchestrator cannot read any workload's state
• Storage growth is bounded by generation count, not by interruption rate

What it costs

• Key management per workload
• RGW capacity is sized for rare bursts and idle most of the time

Choose differently when. Workloads with very large state, tens of gigabytes, cannot write a final checkpoint in 45 seconds on any store and should rely on periodic checkpoints and a lower tier, or a parallel file system near the compute.

Why it holds up over time. S3-compatible object storage, bucket policies and lifecycle rules are supported by every object store the estate is likely to adopt. The ownership rule, the workload's data in the workload's boundary, stays true whoever runs the storage.

Lesson. The platform that moves your work around should not be able to read it.

ADR-20 · The periodic checkpoint interval is set as a loss budget, not a habit

Status: Accepted · Shown on views: 12, 20, 28

How often should work be checkpointed when most reclaims come with notice?

Context. The requirement forbids relying on the final write alone and caps wasted work at 3% of reclaimable hours. With notice, most reclaims end in a final checkpoint; only no-notice terminations, overruns and failed final writes lose work since the last periodic checkpoint. Checkpointing too often wastes compute and storage; too rarely makes the rare loss large.

Decision. The expected unprotected loss rate is computed from the assumptions: 4% reclaimed an hour × 3.5% of reclaims without a successful final write ≈ 0.14% an hour, an effective mean time between unprotected losses of about 714 hours. With a 45-second checkpoint, the Young–Daly interval √(2 × C × MTBF) is about four hours. The default interval is capped at 60 minutes, because the no-notice share is an assumption and one lost hour is the largest loss owners said they would accept. For requeue workloads the compiler enforces rate × unit length ÷ 2 ≤ 3%.

How it works on premises. Interval and unit-length rules live in the compiler. The ledger measures actual wasted work per workload and per cause, and the tier defaults are revisited quarterly against it.

What it buys

• Checkpoint overhead stays well under 1% of compute
• The interval changes when measured rates change, not when someone worries

What it costs

• A no-notice loss costs up to an hour
• Relies on the no-notice assumption being revisited against data

Choose differently when. If no-notice terminations turned out to be 10% of reclaims, the effective MTBF falls to about 250 hours and the interval should drop towards 30 minutes.

Why it holds up over time. The formula is independent of any product and has held since the 1970s. Updating its inputs from the ledger keeps the interval right as hardware, providers and failure rates change.

Lesson. Checkpoint frequency is a budget decision. Write down the budget and the frequency follows.

ADR-21 · Restart semantics are declared, effects are published only on completion, and poison units are quarantined

Status: Accepted · Shown on views: 20, 32, 12

How does a resumed or requeued unit avoid duplicating externally visible effects, and how does the platform stop paying for a unit that never finishes?

Context. Losing capacity may slow a workload down but must never produce a partial or duplicated result. A unit that writes output, sends a message or updates a record before it is reclaimed can do so again after it resumes. Separately, a unit that consistently needs longer than the time between interruptions is re-executed on ever-shorter leases indefinitely.

Decision. Every contract declares restart semantics: idempotent, output-commit or not restartable (not admitted to reclaimable capacity). For output-commit, the client writes results to a staging prefix keyed by unit and attempt, and publishes them atomically only when the unit completes. Each unit carries an attempt counter; after four attempts ended by reclaim it is moved to a quarantine queue and its owner notified, in Phase 2.

How it works on premises. Staging prefixes in the workload's own RGW bucket, with publish as a copy to the final key followed by a completion marker. Spark and PyTorch integrations use their frameworks' commit protocols. Attempt counters on lifecycle events, quarantine as a Kafka topic per partition.

What it buys

• No duplicated result from a reclaim
• A pathological unit costs four attempts, not a month

What it costs

• Output commit adds a copy per unit
• Quarantine needs an owner to look at it

Choose differently when. Workloads whose outputs go to an idempotent store with natural keys can declare idempotent and skip the staging step.

Why it holds up over time. Exactly-once effects through staged output and atomic publish is how every reliable batch system has worked. The rule outlasts any change of storage or framework.

Lesson. A retry is only safe if doing the work twice has been made harmless.

Floor and fallback

How much guaranteed capacity is held, who may spend beyond it, and how fallback and return avoid storms and oscillation.

ADR-22 · The reliability floor is bought, runs real work, and is drawn from committed capacity first

Status: Accepted · Shown on views: 06, 17, 25

Is the reliability floor bought, or achieved by diversification?

Context. A pre-provisioned floor of guaranteed capacity is simple and always there, and is paid for whether used or not. Diversification can achieve a similar effective floor for less, until a correlated event shows the pools were not independent. A third option uses capacity commitments consumed by baseline work in normal times and reallocated under stress. The requirement sets at least 30% of each tier-1 workload's capacity on non-interruptible capacity at all times.

Decision. The floor is bought, but it is not idle. Each tier-1 and tier-4 workload always runs its floor share of units on guaranteed capacity, drawn first from its business unit's committed Blazar reservations, already paid for, and only then from on-demand guaranteed capacity. The floor is spread across at least three halls with no hall above 40%. Diversification is not trusted to substitute for any part of the floor until learned correlation groups have a year of evidence.

How it works on premises. The floor keeper in the controller maintains floor instances in a separate orchestrator-owned project, sized in normalised units, and rebuilds lost floor within 15 minutes of a hall loss from committed capacity in other halls.

What it buys

• Floor capacity produces output every hour it is paid for
• Correlated reclaims cannot take a deadline workload to zero

What it costs

• About €4.3k a month per 10 points of tier-1 floor at planning rates
• Up to 15 minutes below floor after a hall loss

Choose differently when. Once learned correlation groups have enough history to bound the worst correlated week with confidence, some of the floor can be replaced by stricter diversification caps, and the replay in view 06 is the evidence.

Why it holds up over time. Whatever the capacity market, a guaranteed floor is the one reliability mechanism that does not depend on an estimate being right. Making it do useful work keeps it economically defensible as rates change.

Lesson. Insurance that also does work is much easier to keep paying for.

ADR-23 · Fallback restores the floor without asking, spends automatically within a declared cap, and escalates beyond it

Status: Accepted · Shown on views: 05, 22, 31

Is fallback to guaranteed capacity automatic, budget-gated or human-approved?

Context. Automatic fallback protects deadlines and turns capacity loss into cost, which is the trade the platform exists to make, but an unbounded automatic fallback during a broad shortage can multiply the compute bill in an hour when nobody is watching. A budget gate bounds the damage but turns a cost incident into an availability incident when it closes. Human approval alone is safe and too slow. The requirement asks for fallback gated by a spend authority declared in the contract, with escalation beyond it.

Decision. Three steps. Restoring a workload's floor is pre-authorised by its tier and draws on committed capacity, so it is automatic. Fallback above the floor, triggered by an interruption-rate breach, shortfall of five minutes or deadline risk, is automatic up to the contract's 24-hour spend cap, reserved in the ledger before launch. Beyond the cap, the arbiter pages the budget owner named in the contract with priced options; without an answer in 15 minutes it records shortfall and spends nothing more.

How it works on premises. The arbiter in the controller, the spend ledger in PostgreSQL, approval requests through the incident platform, approval by OIDC with WebAuthn, every step to the audit log service.

What it buys

• The worst plausible fallback cost is the sum of declared caps
• A shortage beyond authority is visible as shortfall, not hidden as spend

What it costs

• A deadline can be missed because a cap was too low
• Budget owners must be reachable, or accept that shortfall

Choose differently when. Workloads whose deadline miss costs far more than any compute bill, such as regulatory filings, should carry a high cap and a pre-authorised approver so escalation never gates them.

Why it holds up over time. Spending authority, a pre-authorised baseline and escalation beyond a limit are how organisations have controlled spending for a long time. Encoding them in contracts keeps them stable across changes of price, provider or team.

Lesson. Automation should be allowed to spend money, up to the amount a named person agreed it could.

ADR-24 · Fallback is staggered by priority and return waits for sustained health

Status: Accepted · Shown on views: 22, 21, 32

How are fallback storms and oscillation between purchase models prevented?

Context. When many workloads fall back at once, they all reach for the same guaranteed capacity that guaranteed tenants are also claiming. When capacity recovers, returning immediately can pull the fleet back into a pool that is about to be reclaimed again, producing an oscillation that costs both money and interruptions.

Decision. Fallback launches are limited to 40 instances a minute per region, ordered by deadline risk and then tier. Return starts only when a workload's eligible pools have been healthy for 60 minutes, below half its declared interruption rate, and then moves 20% of fallback capacity an hour, newest fallback first.

How it works on premises. A priority queue in the arbiter with a token bucket shared with the acquirer's regional budget; return controller state in PostgreSQL.

What it buys

• Guaranteed tenants are not crowded out by a fallback wave
• The fleet does not flap between purchase models

What it costs

• Lower-priority workloads wait for fallback
• Some fallback spend continues for an hour or more after recovery

Choose differently when. If guaranteed capacity were plentiful and fallback rare, the stagger could be relaxed; the return hysteresis should stay.

Why it holds up over time. Rate limiting and hysteresis are control-theory basics that apply to any system switching between two states under noisy signals.

Lesson. Leave in order, come back slowly.

Evidence and money

How every capacity event is recorded, how the saving is calculated honestly, who sees it, and who is woken when something goes wrong.

ADR-25 · Lifecycle events ride the estate's Kafka and ClickHouse; capacity actions are audited separately

Status: Accepted · Shown on views: 15, 13, 31

How is every capacity event recorded durably at low cost, and why is the audit record kept apart from it?

Context. The requirement asks that every placement, reclamation, drain, checkpoint, fallback and replacement be a durable event retained 400 days, that cost attribution be kept seven years, and that every capacity action be recorded with its causing identity in an audit record independent of the telemetry pipeline. It also caps the orchestrator's own cost at 2% of the saving.

Decision. Agents spool events locally and Vector ships them to a Kafka topic keyed by instance; controllers write through a transactional outbox. ClickHouse keeps events 400 days and cost rollups seven years, and exports cost attribution monthly as Parquet to Ceph with Object Lock. Capacity actions (launch, terminate, fallback, approval) are written from the same outbox, separately, to the estate's audit log service.

How it works on premises. The estate's shared Kafka and ClickHouse clusters with dedicated topics, databases and quotas. Vector on each node. Ceph RGW Object Lock in compliance mode for the finance archive.

What it buys

• Any instance's life can be reconstructed for 400 days
• Finance retention is outside the query store

What it costs

• Dependence on shared clusters' availability, mitigated by node spools
• Events from nodes deleted during long outages can be lost; the provider's deletion event records the loss

Choose differently when. An estate without shared streaming and analytics platforms would be better served by PostgreSQL with partitioned tables at this volume.

Why it holds up over time. Events with causes are the most portable form of history. The storage underneath can change; the events and the reports built from them remain.

Lesson. Keep the record of what the automation did somewhere the automation cannot change.

ADR-26 · The saving is computed net against a guaranteed-rate counterfactual, with every cost attributed

Status: Accepted · Shown on views: 28, 15, 06

How is the saving calculated so that it is true, and who pays for each cost of the discount?

Context. The gross discount, the difference between two rate cards times hours, overstates the saving. Redone work, warm headroom, fallback spend, checkpoint storage and the orchestrator's own footprint are all costs of running on reclaimable capacity. On hardware the organisation owns, there is a further subtlety: a rate-card saving becomes money only if the organisation buys less guaranteed capacity as a result.

Decision. Net saving per workload equals the useful work delivered, valued at the guaranteed rate, minus all spend: reclaimable hours, fallback hours, warm headroom, checkpoint storage and a pro-rata share of the orchestrator. Wasted work is removed from useful work rather than added as a cost, so it is counted once. Losses caused by workload defects are reported to that workload and not charged to reclaim. The planning figure is €199k a month net, 59%, with the orchestrator at 1.6% of the saving. The ledger labels the saving as rate-card and publishes reclaimable hours to capacity planning.

How it works on premises. Hourly attribution jobs in ClickHouse joining lifecycle events, instance hours, rate cards, RGW usage and the orchestrator's own resource usage; Grafana reports per workload; monthly export to finance.

What it buys

• Workloads that do not benefit are identified and recommended off
• The orchestrator's own cost is visible against what it produces

What it costs

• Attribution logic to maintain
• Numbers that are less flattering than a vendor slide

Choose differently when. Where capacity is bought per hour from a public cloud, the rate-card saving is already money, and the procurement caveat disappears.

Why it holds up over time. Counterfactual minus all attributable cost is the definition of a saving, and it does not change with rates, tiers or providers. New costs join the ledger as new lines.

Lesson. A saving that ignores what it cost to obtain is a price comparison, not a saving.

ADR-27 · Showback per workload now; chargeback priced by the declared contract later

Status: Accepted · Shown on views: 12, 28, 29

Who holds the savings, and who holds the risk?

Context. If the workload owner sees the discount, they are motivated to over-declare their tolerance and under-declare their drain budget. If the platform holds the savings and charges a flat internal rate, the incentive to move eligible work onto reclaimable capacity disappears. The requirement asks whether declarations can be validated cheaply enough to make over-declaration self-correcting, and whether a charging model can reward accurate declaration.

Decision. The MVP shows each owner their workload's net saving, including the fallback spend and wasted work their contract caused, and alarms when measured interruption exceeds what they declared. Phase 3 introduces chargeback in which the reclaimable rate paid depends on the declared tier and tolerance: declaring a higher tolerance buys a lower rate but carries the fallback and waste costs of being wrong, so accurate declaration is the cheapest option.

How it works on premises. Showback reports in Grafana from the ClickHouse ledger. Chargeback as a monthly export to the finance system with contract-tier rate multipliers agreed with finance.

What it buys

• Owners see both sides of their choice
• Over-declaration becomes visible and later expensive

What it costs

• Chargeback needs finance agreement and a year of trusted data
• Showback alone relies on owners caring

Choose differently when. In an organisation without internal chargeback at all, showback with executive targets for the eligible population may be the whole model.

Why it holds up over time. Tying price to declared risk is how insurance has always been priced. It stays correct whatever the actual rates become.

Lesson. If declaring the truth is not the cheapest option, the declarations will drift away from the truth.

ADR-28 · Metrics on Prometheus and Thanos, with alarms routed to whoever can fix the cause

Status: Accepted · Shown on views: 27, 23

What is measured, and who is woken for which failures?

Context. The requirement asks for interruption rate, time between interruptions and drain overruns per workload and pool; shortfall reported apart from interruption; alarms when realised saving falls below a threshold or the orchestrator's own cost exceeds its ceiling. An alarm sent to the wrong person is noise: a workload owner cannot fix a bad pool, and the platform cannot fix a workload that over-declared.

Decision. Metrics in Prometheus per region with Thanos for 13-month queries; reports in Grafana over Prometheus and ClickHouse; alerts through Alertmanager to the incident platform. Workload owners are alerted only on breaches of their own contract. The platform team is paged for pool behaviour, drain overrun rates, tier-1 shortfall beyond 15 minutes, control-plane health, bundle age and the orchestrator's cost ratio.

How it works on premises. The estate's Prometheus and Thanos stack with dedicated recording rules and retention; Alertmanager routes by label: contract owner or platform.

What it buys

• Owners hear only about their own promises
• Platform problems page the platform

What it costs

• Routing rules must follow contract ownership changes
• Shared metrics stack availability matters

Choose differently when. A very small estate where the platform team also owns every workload can merge the two routes.

Why it holds up over time. Routing by who can fix the cause survives any change of tooling or team structure.

Lesson. An alert is a request for action. Send it to someone who can take the action.

Security and authority

Who holds the power to acquire, terminate and spend, what a compromised node or workload can reach, and what happens to a reclaimed node's credentials.

ADR-29 · Capacity authority is scoped by OpenStack project, credentials are short-lived, and nodes hold none

Status: Accepted · proof required · Shown on views: 30, 09, 10

Who can launch and terminate reclaimable capacity, and how is a defect or compromise kept from reaching unrelated capacity?

Context. The requirement asks that capacity permissions sit under an identity distinct from any workload, that no workload can terminate or acquire capacity, and that termination authority be scoped to instances the orchestrator launched and tagged. Tag-based conditions in access policy are fragile: tags are mutable, and anyone who can edit them can move an instance into or out of scope.

Decision. Reclaimable and fallback instances live in dedicated OpenStack projects per region and partition, in which only the orchestrator's identity has a role. The orchestrator obtains a project-scoped application credential from OpenBao for one hour at a time. Before any termination the controller also checks that the instance's tags and its database row agree. Node agents and workloads hold no provider credential of any kind.

How it works on premises. Keystone projects and roles; OpenStack application credentials issued by an OpenBao plugin or rotation job; OpenBao policies bound to the controller's SPIFFE identity.

What it buys

• A compromised controller can only affect its own projects
• Workloads cannot acquire or terminate capacity by any route

What it costs

• More projects and quotas to manage
• Credential issuance becomes a dependency of acquisition, not of drain

Choose differently when. A provider with robust, immutable resource ownership in its access model could scope by ownership rather than by project.

Why it holds up over time. Structural isolation holds regardless of how policies are written in future provider releases. The rule, only the orchestrator can act in its projects, is simple to audit.

Lesson. Scope power by where things live, not by labels someone else can edit.

ADR-30 · Floors, spend caps and fallback policy need a second, higher authority

Status: Accepted · Shown on views: 06, 26, 31

Who may change the settings that determine how much the organisation spends on reliability?

Context. The requirement asks that changes to the guaranteed floor, the spend cap and fallback policy need a separate, higher authority than routine contract edits, and that moving a workload from reclaimable to guaranteed capacity be treated as a spend-authorising action subject to audit.

Decision. Routine contract fields need the workload owner's and platform team's review. Changes to floors, spend caps, approvers and fallback policy additionally need a signature from the capacity governance group, and the distributor refuses a bundle whose governed fields changed without it. Approvals of spend beyond a cap require a named approver authenticating with a phishing-resistant factor. Operators of the orchestrator cannot approve their own spend.

How it works on premises. Forgejo CODEOWNERS and branch protection for governed paths; a second cosign signature by a key held for the governance group in OpenBao; OIDC with WebAuthn from the estate identity provider for approvals.

What it buys

• Money decisions have a named, separate owner
• Enforcement does not rely on repository settings alone

What it costs

• Slower floor and cap changes
• A second signing key to protect and rotate

Choose differently when. In a small organisation where the platform lead also owns the budget, a single signature with audit may be proportionate.

Why it holds up over time. Separation between those who operate a system and those who authorise its spending is an organisational control that outlives any tooling.

Lesson. The people who can change a system's behaviour should not be the only ones who can change what it costs.

ADR-31 · A terminated instance's identity and credentials are revoked, not left to expire

Status: Accepted · Shown on views: 30, 18

What happens to the credentials a reclaimed node held?

Context. The requirement asks that a reclaimed instance's credentials and cached secrets be invalidated on termination rather than relying on the instance disappearing. At 4% reclaimed an hour, thousands of nodes a day end with live identity documents and leases that would otherwise stay valid until expiry. A node that is reclaimed but not fully deleted, or whose disk image is retained, is the case that matters.

Decision. On a termination event the controller bans the node's SPIRE agent, so no new SVIDs are issued to it, and revokes OpenBao leases issued to that node's identities by prefix. Node-bound Kubernetes service-account tokens become invalid when the node object is deleted. The agent also revokes its own tokens on a best-effort basis before T+120.

How it works on premises. SPIRE server agent ban API; OpenBao lease revocation by prefix keyed on node identity; Kubernetes node deletion by the cluster's node lifecycle controller.

What it buys

• No credential outlives its node by more than seconds
• The no-notice case is covered centrally

What it costs

• Revocation load on SPIRE and OpenBao during large reclaims
• Revocation is best-effort if those services are down, falling back to expiry

Choose differently when. If credentials were already bound to a hardware key that dies with the node, central revocation would be less important.

Why it holds up over time. Revoking identity when the thing it identifies ends is correct under any identity system.

Lesson. An identity should end when the thing it names ends, not when a timer eventually notices.

Operations

How the platform changes without becoming the cause of the interruptions it manages, and how its promises are tested on an ordinary day.

ADR-32 · Releases ride the fleet's own churn: canary partitions for contracts, new launches for agents

Status: Accepted · Shown on views: 26, 25

How do contract changes and platform code reach thousands of nodes without the rollout becoming a source of interruptions?

Context. Upgrading an agent in place on 5,000 nodes means restarting the component that must not be absent when a notice arrives, and draining nodes to upgrade them would add interruptions to a fleet whose whole point is to handle fewer. But reclaimable fleets turn over constantly by design.

Decision. Contract bundles go to one partition for 24 hours, gated on overrun rate, wasted work and breach alarms, then to all. Controller releases canary on one partition. The agent is never upgraded in place except for urgent security fixes on nodes not draining: a new image is used for 5% of launches, then all, and the fleet turns over through reclamation and scale-down, reaching about 95% within 72 hours at the baseline reclaim rate.

How it works on premises. Forgejo Actions for build and signing; diskimage-builder for images, published to Glance; Argo CD for controllers; image selection per pool in the acquirer.

What it buys

• Rollouts add no interruptions
• Rollback is choosing the previous image

What it costs

• Agent versions coexist for days, so the bundle schema must be backward compatible for two versions
• Rollout speed depends on reclaim rate

Choose differently when. If the fleet were long-lived, with reclaims rare, image-based rollout would be too slow and in-place upgrades with drain-aware ordering would be needed.

Why it holds up over time. Using the system's natural churn as the deployment mechanism works for any ephemeral fleet, and it keeps the version skew rules explicit.

Lesson. If your fleet replaces itself every few days anyway, let that be the rollout.

ADR-33 · Reclamation is exercised daily in production through the provider's own reclaim path

Status: Accepted · Shown on views: 03, 29, 32

How does the platform know its interruption path still works on a day when nothing is going wrong?

Context. The requirement asks for regular deliberate reclamation in production, because an interruption path tested only by real interruptions is tested least when the fleet is healthiest. It places scheduled exercises in Phase 2. Fail-static behaviour, which must be verified by exercise in the MVP, cannot be verified without them.

Decision. An exercise job reclaims 1% of reclaimable nodes per partition each working day through the provider's reclaim API, using the same notice, relay, agent and drain path as a real reclaim, labelled as a drill in lifecycle events. A quarterly game day reclaims a hall's reclaimable capacity with the control plane stopped. This moves into the MVP.

How it works on premises. A CronJob in the management cluster calling the reclaimer's API with the private cloud team's agreement; drill events excluded from pool scoring but included in overrun and drain metrics.

What it buys

• Regressions in the notice path show up within a day
• Fail static is proven, not assumed

What it costs

• About 1% extra churn and its wasted work
• Needs the provider team's standing agreement

Choose differently when. A fleet with very frequent natural reclaims in every pool might get enough exercise without drills, though not of the control-plane-down case.

Why it holds up over time. Exercising failure paths on a schedule is independent of any tool and remains the only way to keep them working through changes in code and staff.

Lesson. A recovery path that runs only during incidents is a recovery path you are testing during incidents.

Every package used, in one table

Every open-source package named on the views, what it is, the job it does in this design, and what was considered instead.