Spot Capacity Orchestrator  ·  View 21 of 32  ·  5 · Runtime

Correlated Reclamation

Guaranteed tenants take back one host aggregate in hall B: 800 nodes, one notice wave, and what each part of the system is doing at each moment.

Editable source SVG draw.io All views
T0 T+2 s T+60 s T+120 s T+3 min T+15 min Provider 800 notices · aggregate 7 Advisory on 2 more 800 instances deleted Guaranteed demand met Node agents Bundles already local 800 drains signalled Most drains finished ≤ 4 overruns counted Control plane Events arrive Group 7 excluded Replace in halls A, C, D Serving ≤ 180 s p99 Fallback where breached Checkpoint store Final writes begin Peak ≈ 10 GB/s Writes committed Restores start Tier-1 workloads Floor untouched Running at ≥ 30% Units resumed Deadline risk rechecked Correlated Reclamation — 800 Nodes in Hall B, Minute by Minute The replacement rate is capped at the region's API token budget. A request storm against Nova would slow every tenant, including the one reclaiming. v 1.0 · owner Compute Platform Architecture · date 2026-09

Why it stays an event, not an outage

  • Every drain was decided before T0, by a bundle already on each node. Nothing central queues 800 decisions.
  • No workload had more than 30% of its units in the aggregate's correlation group, so the worst-hit tier-1 workload lost 30% of its reclaimable share and kept its whole guaranteed floor.

The load nobody sizes for

  • Final checkpoints, not drains, are the peak. About 40% of the 800 nodes run checkpointing units; 320 writes of 1.4 GB inside 45 s is about 10 GB/s into Ceph RGW. The region's gateways are sized for 20 GB/s, and the proof phase measures it (view 29).
  • Replacement is capped by the Nova token budget. A faster replacement would be a request storm against the same API guaranteed tenants are using to claim the capacity back.

Assumption

  • The replacement halls have headroom. When they do not, the shortfall is reported as shortfall, separately from interruption, and the fallback path in view 22 takes over.
◀  Checkpoint and Restore All views Fallback and Return  ▶
Solution Architecture v1.0 · Compute Platform Architecture · 2026-09 · 32 views · open source, fully on-premises21-correlated-reclamation