Spot Capacity Orchestrator  ·  View 27 of 32  ·  6 · Operations

Observability

Five signal families from emission to action, and a rule about who gets woken for each.

Editable source SVG draw.io All views
Emit Collect Store Consume Act Interruption Agent drain events Vector to Kafka ClickHouse · 400 d Overrun rate per pool Contract flagged Capacity Claims · shortfall Prometheus Thanos · 13 mo Shortfall apart from reclaim Page at 15 min, tier 1 Placement quality Pool and group shares Prometheus Rankings · 90 d Blast-radius board Replan on cap breach Money Rate card · spend ledger Attribution · hourly Cost rollups · 7 y Net saving per workload Move-off recommendation Platform health Leader · bundle age Prometheus Thanos · 13 mo Own cost vs saving Page SRE, never workloads Observability — What Is Measured, and Who Is Woken Workload owners are alarmed only on their own contract breach. Pool behaviour pages the platform, because it is the platform's to fix. v 1.0 · owner Compute Platform Architecture · date 2026-09

Decisions

  • Shortfall is measured apart from interruption. They have different causes, lack of capacity versus loss of capacity, and different fixes.
  • Workload owners are alarmed only on their own contract: a measured interruption rate above what they declared, or realised saving below their threshold. Pool behaviour pages the platform.
  • The orchestrator's own cost is a metric with an alarm at 2% of the saving it realises.

SLOs

  • Notice to drain signal ≤ 2 s p99. Overruns ≤ 0.5% of reclamations. Wasted work ≤ 3% of reclaimable hours. Unsatisfied demand ≤ 0.1% of requested capacity-hours a month. Control plane ≥ 99.9%. Node-local handling ≥ 99.99%.

Adopted

  • Prometheus with Thanos for metrics, Grafana over Prometheus and ClickHouse for reports, Alertmanager into the incident platform. Nothing here is built.
◀  Change Pipeline All views Net Savings Ledger  ▶
Solution Architecture v1.0 · Compute Platform Architecture · 2026-09 · 32 views · open source, fully on-premises27-observability