All or nothing, once

Picture a training run on its eleventh day. Five hundred and twelve pods, each holding accelerators that were difficult to get and are expensive to hold, all of them blocked in the same all-reduce. On one node a GPU stops answering. The pod on it dies. The other five hundred and eleven are healthy, scheduled, running, and completely useless: a collective operation with a missing rank is a barrier that never clears. Whatever happens next — a controller notices, a checkpoint is reloaded, someone is paged — the cluster spends the interval burning the most contended resource it owns on a job that cannot advance.

Ask Kubernetes about that group and, if you are running the new gang scheduling API, it will tell you the group's condition is PodGroupInitiallyScheduled=True. This is not a bug. It is not even misleading. The pods were, initially, scheduled.

Gang scheduling has been promoted to beta in the v1.37 cycle, carrying new Workload and PodGroup types in scheduling.k8s.io/v1beta1 behind a GenericWorkload feature gate, with stable targeted for v1.38. It is a genuine piece of work and overdue: the KEP notes that gang scheduling had been implemented outside kube-scheduler at least four times, and this one formally replaces the old coscheduling KEP. With it comes a phrase that has followed batch scheduling around for years — all-or-nothing. That phrase is the reason to read the specification carefully, because what went to beta is all-or-nothing placement, and what the workloads in the motivation section need is all-or-nothing existence. Between the two lies the entire runtime of the job. Core Kubernetes now has an excellent object for the first and still has none for the second.

The API is candid about this, in the way specifications are candid — in field comments. A gang's size is minCount, "the minimum number of pods that must be schedulable or scheduled at the same time for the scheduler to admit the entire group." The comment then adds that minCount "is only enforced during scheduling," so changes to it never affect pods already placed. The status story is tighter still. There is one condition, PodGroupInitiallyScheduled, and once it reaches True the KEP designates it a terminal scheduling state that does not revert. Failed cycles for pods beyond minCount do not regress it. And then, plainly: scheduled pods may later be evicted or affected by node failures, but the PodGroup status will not track those post-scheduling disruptions. A future, separate component might, the KEP says. The observable surface agrees — the beta metrics count scheduling attempts, algorithm durations and preemption victims. None of them describes whether a gang is currently whole.

None of this is oversight, and the project is not confused about what these workloads require. The companion KEP on workload-aware preemption, also beta in v1.37, states the requirement more clearly than most commentary does: tightly-coupled workloads need ongoing communication across their pods not only at startup but across their whole lifetime, so disrupting a single pod effectively disrupts the whole workload even though the rest are still running. That sentence is the argument for everything above. Its authors went further and judged that shipping gang scheduling without workload-aware preemption "does not provide enough value for the end users," which is why the two features were merged under one gate for beta. The scheduler that admits gangs atomically now also preempts them atomically.

Look at how that protection is switched on, though. Preemption granularity is a new DisruptionMode on the PodGroup — Single, meaning members can be disrupted independently, or All, meaning they can only be disrupted together. It is immutable after creation, controllers are expected to fill it by copying from the workload template, and it defaults to Single when unset. The field that stops a gang being pulled apart one pod at a time defaults to permitting exactly that, on an object whose whole purpose is atomicity, filled in by controllers that the gang KEP itself lists as still mid-integration: the Job controller landed in v1.36, while JobSet, LeaderWorkerSet, KubeRay and TrainJob — the controllers that actually run this class of work — are described as ongoing. Defaults are the most consequential architectural decisions a platform makes, because they are what the system does for everyone who did not know there was a choice.

Preemption is also only one way to lose a pod, and the boundaries here are explicit rather than implied. Extending workload-awareness to other kinds of disruption, including hardware failures and kubelet eviction, is a stated non-goal. So is rescheduling a workload that has been preempted. At the scale where gang scheduling earns its keep, hardware failure is not an edge case, it is a Tuesday — and it is the case the new machinery deliberately does not cover.

There is a second gap, quieter and about arithmetic. The preemption algorithm generalises the existing pod-level one: identify victims of lower priority, reprieve what would violate a PodDisruptionBudget on a best-effort basis, then sort what remains by importance, which is tentatively priority first and, within a priority, pod groups ahead of individual pods. For a pod-group preemptor the search domain starts as the whole cluster. It is a defensible design, honestly labelled as a simple one. But nothing in it knows how long a victim has been running. Preempting a training gang eleven days into a fourteen-day run and preempting one that started ninety seconds ago are, in the scheduler's accounting, the same act at the same price. The real cost is accelerator-hours of unrecoverable progress since the last checkpoint, multiplied by the width of the gang, and that quantity does not appear anywhere in the calculation. We have standardised a cost function that is uncorrelated with the cost.

The strongest objection to all of this is that it is simply correct layering, and it deserves to be taken at full strength rather than waved at. A scheduler's job is admission. It decides where things go; it does not keep them alive. Dragging kube-scheduler into the control loop of every pod failure in every large gang would make the most contended component in the cluster responsible for the noisiest events in it, and the KEP's own reasoning points the other way — PodGroup was decoupled from Workload precisely to escape etcd object-size limits and read-modify-write contention on a central object. Runtime membership belongs in a controller above, and it is already there. Kueue has offered waitForPodsReady since version 0.3.0: it watches a workload until all its pods are ready and, if they are not within a timeout, evicts and requeues the whole thing. It has a recoveryTimeout for the exact scenario this piece opened with — a workload already running with a pod gone not-ready — and its documentation says why, in the language of someone who has paid for it: the workload often cannot progress, leading to wasted resources. Its condition is PodsReady, and unlike the core one it can be False. That page was last revised three days ago, tightening timeout defaults and adding an alpha unscheduledTimeout. The runtime half of the problem is being actively worked. It is just not being worked in core.

So the division of labour is real, and the people who drew it were right. Everything above is read from the specifications and the two projects' own documentation rather than from a production fleet, which is the limit of this argument — and also the reason the wording carries so much weight, because those documents are the only thing most teams will ever consult. What follows from them is still uncomfortable, for two reasons.

The first is a vocabulary collision. Kueue's page is titled "Setup All-or-nothing with ready Pods." The core feature is documented as all-or-nothing scheduling. Both projects use the phrase; they mean different intervals by it; and the one now blessed as a core API with a conformance-bearing feature gate is the weaker of the two. Specifications propagate faster than their footnotes. A year from now, "we're on native gang scheduling" will be said in design reviews to mean something the API has never claimed.

The second is that these layers do not compose by themselves. minCount is a scheduling constraint in core, deliberately mutable so elastic jobs can scale. Kueue's timeouts are controller policy. A framework's world size is a third number, living in the training code. Set minCount below the world size your collective actually requires and core Kubernetes is fully satisfied — the gang was admitted, the condition is True — while your job hangs on a barrier. Three components hold three versions of "how many is enough" and nothing reconciles them. That is not a flaw in any one of them. It is the space between them, and the space between components is where systems fail and where no postmortem has an obvious owner.

Which leaves the interesting question, one worth carrying beyond Kubernetes: the useful thing to know about a guarantee is rarely how strong it is. It is how long it lasts. We are fluent at writing predicates — this holds, that is satisfied, the group was admitted — and nearly mute on tense. SLOs get windows because a decade of operational pain forced them to. Schema contracts, attestations, admission decisions and readiness conditions mostly do not, and so they are read as standing promises when they were only ever observations with a timestamp.

The Kubernetes authors, to their credit, wrote the tense into the name. The condition is not called PodGroupScheduled. It is called PodGroupInitiallyScheduled, and that adverb is carrying the entire warranty expiry, in a field most people will read once. The API is telling the truth about what it knows. The question is whether anyone reading a dashboard at three in the morning will hear it.