Spot Capacity Orchestrator  ·  View 09 of 32  ·  3 · Structure

Control Plane Containers

One region's control plane: signals computed on two clocks, one leader per scheduler-cluster partition, and state that the provider can rebuild.

Editable source SVG draw.io All views
Region management cluster · Kubernetes on guaranteed capacity · 3 halls Signals Pool scorer CronJob · 5 min Exclusion watcher stream · ≤ 60 s sco-controller · leader per partition Demand intake Kubernetes · Slurm adapters Portfolio planner incremental Fallback arbiter floor keeper Acquirer and replacer quota · breakers State Contract distributor 3 replicas Fleet state PostgreSQL · Patroni Pool history ClickHouse Lifecycle events Kafka · estate cluster Provider credential OpenBao Nova · Ironic · Placement private cloud APIs history consume plan generation app credential batched create Control Plane — One Region, Partitioned by Scheduler Cluster Application we own Interface / broker Data store Queue / topic Security / platform External / third party synchronous event / async A partition is one scheduler cluster in one region. Adding a region adds partitions; nothing reconciles the whole estate in one loop. v 1.0 · owner Compute Platform Architecture · date 2026-09

Decisions

  • One controller binary, sharded by partition, with a leader per partition elected through Kubernetes leases. A partition is one scheduler cluster in one region, so a bug or a stuck loop affects one cluster's capacity, and adding a region never grows a loop.
  • Level-triggered reconciliation rather than a workflow engine. The provider's view of what exists is authoritative, so each cycle compares desired plan with observed instances and acts on the difference; there is no in-flight workflow state to lose or replay.
  • Scoring runs every 5 minutes as a batch job; exclusion runs continuously on the event stream. Two clocks, because a pool going bad cannot wait for the next ranking.

Numbers

  • Placement decision ≤ 10 s p99. Exclusion within 60 s of the threshold. Control plane ≥ 99.9% monthly. State RPO ≤ 30 s (synchronous replica makes it effectively 0), RTO ≤ 10 min.

Risks

  • The provider credential is the most powerful secret here. It is fetched from OpenBao for an hour at a time and can act only in the orchestrator's own OpenStack projects (view 30).
◀  Two Planes All views Node Agent  ▶
Solution Architecture v1.0 · Compute Platform Architecture · 2026-09 · 32 views · open source, fully on-premises09-control-plane-containers