Spot Capacity Orchestrator  ·  View 19 of 32  ·  5 · Runtime

Four Drain Paths

The same agent and the same deadline drive Kubernetes pods, Spark executors and Slurm jobs through each scheduler's own machinery, and around it when the scheduler cannot answer.

Editable source SVG draw.io All views
Notice · ≤ 2 s Stop new work Signal the work Preserve Verify At T+120 Kubernetes pods Budget from bundle Taint NoSchedule Evict · grace = budget preStop final write Pods gone from node Job controller retries Spark on Kubernetes Budget from bundle Taint NoSchedule Executor decommission Shuffle to fallback store Blocks migrated Stage continues Slurm jobs Budget from bundle Node state DRAIN USR1 to job cgroup Application checkpoint Job steps exited Requeued, resumes Scheduler API unreachable 800 ms timeout Scheduler not told kubelet graceful shutdown Same final write Outcome spooled Scheduler learns late One Agent, Four Drain Paths — Through the Scheduler, or Around It The agent decides and keeps the deadline. The scheduler's own eviction does the work when it answers in time, so two systems never evict the same pod. v 1.0 · owner Compute Platform Architecture · date 2026-09

Who owns the drain

  • The agent owns the trigger and the deadline; the scheduler owns the mechanics. Driving eviction through the scheduler reuses semantics workloads already understand and avoids two systems evicting the same pod.
  • When the scheduler API does not answer within 800 ms, the agent stops asking. On Kubernetes it takes a systemd shutdown inhibitor and lets kubelet's graceful node shutdown terminate pods by priority, locally. On Slurm it signals the job's cgroup directly. The scheduler learns afterwards, and that is acceptable because the node is going anyway.

Work the platform implements once

  • Spark: executor decommissioning with shuffle and cached-block migration to the Ceph fallback store. Kubernetes Jobs: a pod failure policy that does not count a disruption against the retry limit. Slurm: jobs submitted requeueable with a USR1 signal ahead of termination.

Risk

  • A PodDisruptionBudget can block eviction indefinitely. The agent honours it until the drain budget is spent and then deletes with the remaining grace, because the budget cannot outlive the node.
◀  Hard Notice to Drain All views Checkpoint and Restore  ▶
Solution Architecture v1.0 · Compute Platform Architecture · 2026-09 · 32 views · open source, fully on-premises19-drain-by-scheduler