Spot Capacity Orchestrator  ·  View 24 of 32  ·  5 · Runtime

Pool Exclusion Cycle

How a misbehaving pool is taken out within a minute, kept out long enough to matter, trialled at low exposure, and returned without anyone deciding to.

Editable source SVG draw.io All views
Eligible ranked every 5 min Tripped 3 reclaims in 30 min Excluded ≤ 60 s · no new units Cool-off 2 h, doubles on repeat Probation ≤ 5% of any workload Restored back-off reset after 24 h One pool threshold crossed planner skips it timer starts capped trial 1 h clean next ranking Pool Exclusion — Out Fast, Back Slowly, Never Permanently Application we own Risk / gap Decision point Security / platform Opportunity v 1.0 · owner Compute Platform Architecture · date 2026-09

Thresholds

  • Tripped by three reclamation waves in 30 minutes, where a wave touches at least 5% of the pool's instances, or by acquisition thrash: instances lost within 15 minutes of launch, three times in an hour.
  • Cool-off is 2 hours and doubles on each repeat within a day. Probation caps the pool at 5% of any workload for an hour. The back-off resets after 24 clean hours.

Decisions

  • Exclusion is automatic and never permanent. A permanent exclusion list only grows, and a year later the fleet is concentrated in the few pools nobody has excluded yet.
  • Exclusion is per pool across all workloads, because the cause, guaranteed demand in that pool, is not specific to any workload.

Risk

  • A correlated group can trip one pool at a time and stay ahead of exclusion. The group-level cap and, in Phase 2, learned correlation groups close that gap.
◀  Control-Plane Outage All views Deployment Architecture  ▶
Solution Architecture v1.0 · Compute Platform Architecture · 2026-09 · 32 views · open source, fully on-premises24-pool-exclusion-cycle