Ephemeral Storage Eviction

One pod on a node writes verbose logs into a scratch volume. The node's filesystem crosses its threshold, and within a minute several unrelated services on that node are returning connection resets.

The sequence is deliberate and documented. The kubelet detects node-filesystem pressure - 10% available by default - taints the node so the scheduler sends it nothing new, garbage-collects unused container images to recover space, and then begins evicting pods, selecting by quality-of-service class and by how far each pod exceeds its request for the pressured resource. BestEffort pods go first, then Burstable ones in order of overage.

The kubelet is defending the node, not punishing the culprit. Without per-container ephemeral-storage requests and limits, there is nothing in the data it has that identifies who filled the disk, so the blast radius of one noisy pod is every workload sharing the node.

Why it matters

It is the clearest example of a shared, non-compressible, invisible resource in a container platform. Teams reason carefully about CPU and memory because the fields are in every template, and treat disk as infinite because nothing in their manifest mentions it.

The timing makes it worse. A log-chatty service is chattiest when traffic is highest, so the eviction lands during the peak and takes out neighbours that were handling their own load perfectly. And because eviction recovers the space, the node heals and nobody investigates - the incident recurs at the next peak.

Implementation patterns

• Declare ephemeral-storage requests and limits per container. With a limit set, the kubelet evicts the pod that exceeded its own limit instead of a lottery of neighbours. One field converts a node-wide incident into one pod restarting.
• Do not write logs to a volume. Write to stdout and let the container runtime rotate with a configured size cap, which is bounded by design and collected by the platform.
• Separate the container runtime's filesystem from application scratch space, so image and container storage cannot be consumed by a misbehaving workload.
• Bound temporary files explicitly - a sizeLimit on an emptyDir, or a memory-backed emptyDir where the data is small and the write rate matters, remembering that memory-backed volumes count against the pod's memory limit.
• Alert on nodefs.available and per-pod ephemeral usage before the threshold, and graph eviction counts per node, because a repeated count on one node is the signal rather than a sustained alert.

Industry example

Kubernetes' own node-pressure eviction documentation specifies the thresholds, the taint, the image garbage collection and the ranking by QoS class and overage. The second-order effect it names is the one that surprises operators: evicted pods must pull their images again when rescheduled, and the images were just deleted by the garbage collector, so recovery produces a burst of registry pulls - which can itself be rate-limited exactly when capacity is needed. Public registries have published pull-rate limits since 2020, which turns that burst into a hard dependency on a third party during an incident.

The practical lesson from platform teams who have hit this is that the fix is a default in the template, not a lesson learned per service: ephemeral-storage limits belong in the base manifest the way memory limits already do.

Failure scenarios

• Innocent neighbours evicted mid-request, producing errors in services with no relationship to the cause.
• Image pull storms during recovery, throttled by a registry, extending a one-minute problem into several minutes.
• A memory-backed emptyDir filling, which is charged to the pod's memory limit and therefore shows up as an out-of-memory kill rather than a disk problem - a confusing diagnosis.
• A sidecar writing logs to the same volume as the application, doubling the rate and making attribution harder.
• Eviction loops, where the rescheduled pod lands on another node and fills that one too, walking the problem around the cluster.

Trade-offs

Setting ephemeral-storage limits costs an estimate nobody has: teams do not know how much scratch space their service uses, and an incorrect limit evicts a healthy pod. The mitigation is to measure first, then set a limit with generous headroom - the aim is attribution, not tight packing.

The broader trade is isolation against utilisation, the same one that governs every shared resource. Dedicated node pools remove the blast radius entirely and cost idle capacity; shared nodes are efficient and need every noisy-neighbour control configured. There is no third option where workloads share a disk and cannot affect each other.

When not to use it

On a single-tenant node pool running one application, ephemeral-storage limits add configuration for little benefit - the blast radius is already that application, and the useful control is log rotation. The same applies to a batch cluster where a restarted job is unremarkable.

The control earns its place in proportion to how many unrelated workloads share the node, which is the general rule for noisy-neighbour defences. What should never be skipped, at any tenancy level, is bounding how much a container can write: the limit is about who gets blamed, and the bound is about whether it happens at all.

Interview question

Q: A shared Kubernetes node starts evicting pods during peak traffic and the affected services have nothing in common. Walk me through what happened, what you would change in the next hour, and what you would change in the platform's defaults.

What a strong answer covers: node-filesystem pressure, the taint, image garbage collection and eviction ranked by QoS class and overage rather than by culpability; identifying the writer from per-pod ephemeral usage; the immediate fix of log rotation or stdout logging; ephemeral-storage requests and limits as the attribution mechanism and as a platform default; the image pull storm during recovery and the registry rate limit behind it; and alerting before the threshold with eviction counts per node as the recurrence signal.

Quick check

Quiz: Why can a pod that did not fill the disk be the one evicted? Because node-pressure eviction ranks candidates by quality-of-service class and by overage against their requests, and without ephemeral-storage limits there is nothing identifying the pod that consumed the space.

Flashcard: Which single field changes the blast radius of a disk-filling pod from the node to the pod? A per-container ephemeral-storage limit - the kubelet then evicts the container that exceeded its own limit instead of selecting neighbours by QoS class.