Spot Capacity Orchestrator  ·  View 10 of 32  ·  3 · Structure

Node Agent

What runs on every reclaimable instance, what it holds before work arrives, and the two ways it drains: through the scheduler, or around it.

Editable source SVG draw.io All views
One reclaimable instance Held before any work lands Launch user-data cloud-init · contract subset Contract bundle signed · never expires sco-agent · static Go · systemd Notice watcher stream · 5 s poll Drain executor cordon · signal · evict Drain verifier done or overrun Event spool 64 MB on disk Scheduler and workload kubelet or slurmd graceful shutdown Checkpoint client agent socket Workload process restart-safe entry Notice relay NATS · provider Scheduler API API server · slurmctld Vector to Kafka when reachable notice drain budget local fallback taint · evict on_notice() drain events Node Agent — Everything the Notice Path Needs, Already on the Node Security / platform Application we own Data store External / third party Queue / topic event / async synchronous failure / alternate The agent holds no provider credential. It can cordon and drain its own node; it cannot launch, terminate or reach another node. v 1.0 · owner Compute Platform Architecture · date 2026-09

Decisions

  • A static Go binary under systemd, baked into every image, not a DaemonSet. It must run on Slurm nodes as well as Kubernetes nodes, and it must not depend on the scheduler it may have to work around.
  • The contract subset arrives in cloud-init user-data at launch, and the signed bundle refreshes every 60 s afterwards. A node never has a window where work is running and the drain budget is unknown.
  • Notice arrives on two channels: a subscription to the per-hall notice relay, and a 5-second poll of instance metadata in case the stream is down. A third channel is absence: the no-notice path is covered by leases and periodic checkpoints, not by the agent.

Scope of power

  • The agent authenticates with a node-bound identity, and an admission policy lets it taint and evict on its own node only. It holds no provider credential: it cannot launch, terminate or touch another node, so a compromised agent is a compromised node and nothing more.

Footprint

  • Budget of 20 millicores and 40 MiB per node. At 5,000 nodes that is 100 vCPU fleet-wide, and it counts against the orchestrator's 2% cost ceiling (view 28).
◀  Control Plane Containers All views Integration Surface  ▶
Solution Architecture v1.0 · Compute Platform Architecture · 2026-09 · 32 views · open source, fully on-premises10-node-agent