Spot Capacity Orchestrator  ·  View 32 of 32  ·  7 · Assurance

Failure Modes

The thirteen failure classes from the requirement, each with the signal that detects it, the mechanism that contains it and the path that recovers from it.

Editable source SVG draw.io All views
Detected by Contained by Recovered by Single-pool reclaim Exclusion watcher ≤ 15% per pool Replace in other groups Correlated reclaim Group reclaim rate ≤ 30% per group Floor holds · fallback No-notice termination NotReady · lease expiry Periodic checkpoint Unit requeued Drain overrun Agent at T+118 s Counted · contract flagged Shorter interval proposed Checkpoint store down Client write error Lease released, recorded Resume from last generation No capacity at any price Shortfall ≥ 5 min Floor already running Fallback within authority Fallback storm Arbiter queue depth 40 per min, by priority Floor was pre-provisioned Acquisition thrash Lost within 15 min Back-off doubles Probation return Quota exhaustion Headroom ≤ 20% Alarm before it binds Quota raised by cloud team Price inversion Rate ≥ 85% guaranteed Not a discount pool Move-off recommended Control-plane outage Leader lease lost Fail static Read-only rebuild first Poison work item Attempts ≥ 4 Quarantine queue Owner notified Hall loss Provider hall status Hall excluded · caps rescaled Floor rebuilt ≤ 15 min Failure Modes — Thirteen Classes, Each Detected, Contained, Recovered Every row is exercised: the daily drill covers the first four, the quarterly game day the hall loss and the control-plane outage together. v 1.0 · owner Compute Platform Architecture · date 2026-09

Reading the matrix

  • Containment is designed in before the failure: caps before correlated loss, periodic checkpoints before no-notice loss, a pre-provisioned floor before a fallback storm. Recovery is what the platform does afterwards, and none of it requires the failure to have been foreseen in detail.

The ones that depend on assumptions

  • No-notice termination assumes 2% of reclamations. Hall loss assumes committed capacity exists in the other halls to rebuild the floor. Quota exhaustion assumes the private cloud team raises quotas within a working day. Each is monitored, and each has a card on its own view.

Exercised

  • The daily 1% drill covers single-pool reclaim, drain, overrun detection and no-notice handling. A quarterly game day takes a hall's reclaimable capacity and the control plane down together, because those two are most likely to fail together.
◀  Spend Authority All views
Solution Architecture v1.0 · Compute Platform Architecture · 2026-09 · 32 views · open source, fully on-premises32-failure-modes