Spot Capacity Orchestrator  ·  View 18 of 32  ·  5 · Runtime

Hard Notice to Drain

One reclaimed node, from the reclaimer's notice to deletion at T+120, with the replacement requested in parallel and the drain verified rather than assumed.

Editable source SVG draw.io All views
Reclaimer Node agent Scheduler API Workload Checkpoint bucket Lifecycle events Acquirer 1. T0: terminate at T+120 (via relay) 2. bundle on disk: budget 110 s, SIGTERM 3. taint NoSchedule (800 ms, else local) 4. T+1.4 s: evict · grace 110 s · deadline file 5. notice_received 6. replace from another group 7. final checkpoint 1.4 GB 8. generation 18 committed 9. exited 0 at T+71 s 10. verify: no pods, no job steps 11. drained · 49 s spare 12. T+120: delete instance 13. if still running at T+118: overrun Hard Notice — 120 Seconds, Spent on the Node The acquirer's replacement runs in parallel with the drain and waits for nothing on the node. The deadline is the provider's and is never extended. v 1.0 · owner Compute Platform Architecture · date 2026-09

The order that matters

  • Mark unschedulable, stop routing new work, then signal the workload, in that order, so drain never races new arrivals. The taint is applied before eviction even when the API is slow, falling back to the local path at 800 ms.
  • Verification is an observation, not a return code: no pods and no job steps left on the node. A node still running work at T+118 s is recorded as an overrun, a distinct event that flags the contract as one the workload cannot honour.

Numbers

  • Notice to drain signal 1.4 s here against a 2 s p99 budget. Final checkpoint 1.4 GB in 31 s against 45 s p99 for 2 GB. Overruns budgeted at ≤ 0.5% of reclamations.

What is never done

  • The drain is never extended past the provider's deadline. The deadline belongs to a guaranteed tenant who was promised capacity; the platform's job is to finish inside it or lose the work knowingly and count it.
◀  Placement Decision All views Four Drain Paths  ▶
Solution Architecture v1.0 · Compute Platform Architecture · 2026-09 · 32 views · open source, fully on-premises18-notice-to-drain