Spot Capacity Orchestrator · View 08 of 32 · 3 · Structure
The critical decision
- Centralising the interruption response is tempting because every choice in it is better made with global knowledge. It also puts a round trip to a control plane inside a 120-second window, for thousands of nodes at once, during the event most likely to be degrading that control plane. It works in the steady state and fails exactly when it is needed.
- So the planes split on the interruption boundary. Placement, scoring, acquisition and fallback are global and deliberative; their loss costs convergence. Cordon, drain, checkpoint trigger and lease release are local and immediate; they run from state the node already holds.
Budgets
- Notice to drain signal ≤ 2 s at p99, leaving ≥ 115 s to the workload. Relay delivery ≤ 1 s; decision from the local bundle in milliseconds; scheduler API call capped at 800 ms before the local path takes over.
- Node-local handling ≥ 99.99%, and it works with the control plane unavailable for 12 hours or more.
What the node does not decide
- Replacement, fallback, pool choice or spend. A node that has lost its control plane drains exactly as it always does, and does nothing else.
