Spot Capacity Orchestrator  ·  View 23 of 32  ·  5 · Runtime

Control-Plane Outage

What happens to running capacity, drains, new demand, replacement and evidence as the control plane stays down for minutes, hours and more than a day, and how it comes back.

Editable source SVG draw.io All views
0 – 5 min 5 min – 1 h 1 – 12 h Beyond 12 h On recovery Running capacity Untouched Untouched Shrinks as reclaimed Converges to floor No mass terminate Notice and drain Node-local as always Node-local as always Node-local as always Bundle age reported Bundles refreshed New demand Waits in scheduler Pending work grows Tier-1 shortfall incident Throughput at floor Tier 1 served first Replacement and fallback Paused Paused Floor already paid for Manual floor via runbook ≤ 5% of fleet per 5 min Evidence Spooled on nodes Spooled on nodes Lost with reclaimed nodes Gap marked, not zero Rebuilt from provider Control-Plane Outage — The Fleet Keeps Running Exactly as It Was Recovery starts read-only: two reconcile cycles rebuild state from provider tags before the first action is allowed. v 1.0 · owner Compute Platform Architecture · date 2026-09

Fail static

  • Nothing running is touched: no mass termination, no mass acquisition. Drain continues node-locally from bundles that never expire, because acting on a stale drain budget is far safer than not draining.
  • Reclaimed capacity is not replaced. Throughput therefore decays towards the guaranteed floor over hours, which is what the floor is for.

Recovery is gradual

  • The controller comes back read-only and rebuilds state from provider tags for two cycles. It then acts at no more than 5% of a partition's fleet per 5 minutes, tier 1 first, inside the same API token budget. A control plane that returns and immediately reconciles a large drift is indistinguishable from an outage of its own.
  • An instance that looks orphaned is terminated only if its scheduler reports it idle across three observations, never on the first sighting.

Price stated

  • A tier-1 shortfall beyond 15 minutes is an incident even though the design is working as intended. After 12 hours the fleet is running close to its floors, and the runbook adds floor capacity by hand.
◀  Fallback and Return All views Pool Exclusion Cycle  ▶
Solution Architecture v1.0 · Compute Platform Architecture · 2026-09 · 32 views · open source, fully on-premises23-control-plane-outage