Spot Capacity Orchestrator  ·  View 15 of 32  ·  4 · Data

Lifecycle Event Pipeline

How a drain outcome on a node that is about to vanish becomes a classified, costed event in the ledger and the finance archive.

Editable source SVG draw.io All views
Emit Node agents local spool Controllers transactional outbox Provider notifications reclaim · delete Transport Vector on each node sco.lifecycle Kafka · keyed by instance Classify Exclusion watcher sliding window Reclaim or defect? notice · exit code Store Event history ClickHouse · 400 d Cost rollups per workload · 7 y Act and report Breach alarms Alertmanager Workload reports Grafana Finance export Ceph Object Lock on reconnect monthly Lifecycle Events — From a Node's Spool to the Finance Archive Application we own External / third party Interface / broker Queue / topic Decision point Data store Security / platform event / async batch An event without a cause is not written to the ledger. Defects are charged to the workload's own report, reclaims to the platform's. v 1.0 · owner Compute Platform Architecture · date 2026-09

Decisions

  • Agents write to a bounded 64 MB local spool before anything else, and Vector ships it when the network allows. The spool dies with the node, which is why the outcome is flushed within the last three seconds of the window.
  • Controllers write events through a transactional outbox in PostgreSQL, so a placement and its event commit together or not at all.
  • Classification happens before storage. A unit that exited non-zero with no notice on its node is a defect; one that ended within a notice window is a reclaim; operator drains and drills are labelled at source.

Adopted

  • Kafka and ClickHouse are the estate's shared clusters with dedicated topics and databases. At 30,000 events on the worst day, dedicated clusters would breach the orchestrator's 2% cost ceiling on their own.

Risk

  • Events from nodes deleted during a long control-plane or network outage are lost with the node. The provider's deletion notifications fill the gap with the fact of loss, and the ledger marks the missing outcome as unknown rather than as zero.