Unhealthy does not evict anything

There is a type in the Kubernetes API called ResourceHealthStatus, and it has exactly three permitted values. Healthy: "operates as normal". Unhealthy: "reported unhealthy". Unknown: "The status cannot be determined." They were argued over, written down, given doc comments, and attached to an identifier scheme that resolves a DRA device as <driver name>/<pool name>/<device name>. They surface in pod status, next to a Message field offering "additional human-readable context". They are the only words Kubernetes has ever agreed on for describing whether an accelerator is working.

None of them can stop a pod.

The thing that can stop a pod is a different string entirely, and Kubernetes declines to say what any of its values mean. On 26 August, v1.37 — Garhwal — shipped, and the changelog records it in one line: Promoted the DRA Device Taints and Tolerations feature to GA, making it available via the resource.k8s.io/v1 API. The feature gates behind it, DRADeviceTaints and DRADeviceTaintRules, both flipped to Default: true at 1.37, with a note in the source that they become locked to that default in 1.38. This is not an opt-in. Any cluster on v1.37 running a DRA driver that publishes taints is already honouring them.

This site looked at the other end of the same API eleven days ago — the request side, where an accelerator is still asked for as an integer. This is the response side, and it runs the other way: the device now has something to say back, and what it says can delete you. Everything below is read out of Kubernetes' own repositories — the changelog, the enhancement proposals, the API types, the feature-gate table — with no outside reporting to corroborate it, because this story exists nowhere else yet.

The mechanism is a careful copy of node taints. A DeviceTaint carries a Key, an optional Value, an Effect and a TimeAdded. The effects are None, NoSchedule and NoExecute — PreferNoSchedule is explicitly not valid, and consumers "must treat unknown effects like None". DRA drivers publish taints on devices in their ResourceSlices; cluster administrators add them from the other direction with a DeviceTaintRule, and the two sets are cumulative. A new device-taint-eviction-controller in the controller manager watches for the result. Where it finds a NoExecute taint that nothing tolerates, it sets a DeletionTimestamp; the kubelet stops the containers; the claim controller then drops the pod from ReservedFor and deallocates.

Tolerations do not live on the pod. They live on the request inside the ResourceClaim, which is a deliberate choice — it keeps every pod sharing a claim consistent — and it sets the blast radius accordingly. The API's own wording: "should any of the allocated devices get tainted with NoExecute after allocation and that effect is not tolerated, then all pods consuming the ResourceClaim get deleted to evict them." Not the pod on the bad device. Every pod on the claim. For a multi-node training job holding one claim across a set of accelerators, that is the job.

So the consequential vocabulary is the taint key. Here is what Kubernetes says about it: "The semantic of the value associated with a taint key is defined by whoever publishes taints with that key." Drivers are advised to use their own domain as a prefix to avoid collisions; kubernetes.io/... is reserved for the project. And then the forward-looking sentence, in the conditional: "If some common patterns emerge, then Kubernetes could standardize the name, value and data for certain taints."

Could. At general availability, with the gates on by default, the number of device taint keys that Kubernetes defines is zero.

Compare the file the design was copied from. well_known_taints.go in core/v1 has been in the tree since 2019 and defines eight keys — not-ready, unreachable, unschedulable, memory-pressure, disk-pressure, network-unavailable, pid-pressure, out-of-service — each with a comment stating the condition under which it is added and the condition under which it is removed. That is what made node taints usable by people who do not work on Kubernetes. You could write a toleration against node.kubernetes.io/unreachable knowing what it meant, because the project had committed to a meaning.

The second half of that commitment matters more, and it has no counterpart here at all. Kubernetes ships an admission plugin, DefaultTolerationSeconds, which injects a toleration for not-ready:NoExecute and unreachable:NoExecute into every pod that does not already carry one, with tolerationSeconds set to 300. The project made a policy decision on behalf of every workload in the world: five minutes of node trouble before we start deleting things. Nothing does this for device tolerations. A ResourceClaim arrives at the API server with whatever tolerations its author typed, and the KEP is candid about the consequences: admission policies "could be deployed to limit which tolerations may be used, but as taints are not defined by Kubernetes itself, none of that is part of Kubernetes itself."

What is left is a dial with the middle worn off. Leave tolerationSeconds unset and, in the API's words, you "tolerate the taint forever (do not evict)". Set it to zero or below and you "evict immediately". Write no matching toleration at all and the effect simply applies. The two positions that require no thought are never leave and leave at once; the position in between requires the workload's author to name, in advance and in seconds, how long to endure a fault class the project has not named. Meanwhile the deletion consults no disruption budget. The API comment is explicit that a NoExecuteWithPodDisruptionBudget effect, one that would "honor the pod disruption budget instead of simply deleting pods", is "a possible future new effect" and "currently undecided".

This is where the AI-shaped part of the problem sits, and it is not rhetorical. Node taints were designed in a world where the state worth protecting was somewhere else — in a volume, a database, a queue. What makes an accelerator worth scheduling is precisely the state that lives on it: weights resident in device memory, a warm cache built over hours of traffic, a collective whose ranks were negotiated at start-up. Eviction is cheap when a replica is fungible. Here the thing being reclaimed and the thing being destroyed are the same object. DRA exists because of these workloads; they are the ones for whom the copied semantics fit worst.

The case against reading any of this as a failure is strong, and it is the oldest argument in the project. Kubernetes provides mechanism; vendors provide policy. Only the driver author knows what a pending row-remap or a degraded NVLink actually implies, and a project-level enum of hardware faults would be wrong within a year, parochial to one vendor's silicon, and a liability the moment a new class of device arrives. Nor is the design careless about blast radius: the KEP names the disaster directly — a bad selector "quickly evicting all workloads using any device in the cluster" — and offers the administrator a dry run, publishing the rule with Effect: None, reading the EvictionInProgress condition to see what would happen, then editing it to bite. That is a thoughtful mitigation, and the separation of mechanism from policy is a large part of why Kubernetes is still here.

I think the counterargument is right about enumeration and wrong about abdication, and node taints are the proof. The project never tried to enumerate every way a machine can fail. It defined the small set that carries consequence, gave those a meaning and a default, and left everything else to whoever wanted to invent it. Device taints took the syntax and left that part behind.

And the clinching detail is that Kubernetes did write a device-health vocabulary in this same period — it simply wired it to nothing. Healthy, Unhealthy, Unknown reached beta in v1.36 and are on the way to stable, and the KEP that defines them says plainly that the reporting path is all they are: "The DRA plugin remains responsible for other actions, such as tainting." The split is deliberate, and it is upside down. The words the project was willing to standardise are the ones that only describe. The words that evict were left to the market. Worse, the standardised set is admittedly too blunt to carry the decision anyway — Unhealthy is documented as a temporary condition "since we do not have a mechanism today to distinguish temporary and permanent issues", with PermanentlyUnhealthy filed under future work. Temporary versus permanent is the whole question. Should this job wait, or move? The vocabulary that cannot express it is the one with the doc comments.

Even the targeting is coarser than the problem. A DeviceTaintRule matches on device names, and core DRA made a conscious decision that a device name is not tied to one particular piece of hardware, so that devices can be hot-swapped. The name is a slot. Selecting by attribute is, in the KEP's words, "not possible", because an allocated device's attributes may no longer be published anywhere. A vendor who wants to say this card, this incarnation has to taint from inside the ResourceSlice. The administrator's instrument is the blunter one — and there is no kubectl taint devices; a kubectl command is an explicit non-goal.

The general lesson is about what shared infrastructure finds easy to agree on. A descriptive enum costs a consensus about words. An operative enum costs a consensus about consequences — about whose job dies, and who is answerable when the driver was wrong. Standards bodies and open-source projects will reliably ship the first and defer the second, and the deferral does not leave a vacuum. It leaves the definition to whoever ships the driver, which is to say the party selling the hardware, whose interest in declaring a device degraded is not identical to yours. Kubernetes v1.38's enhancements freeze falls on 30 September. Nothing in flight defines a taint.

None of which will be visible when it happens. The claim will be deallocated, the pods will be gone, and the pod status will carry a ResourceHealth entry reading Unhealthy, with a Message field holding whatever human-readable context the driver cared to supply. It will be accurate. It will have had nothing to do with it.