On a dispatch platform of Swiggy's kind, one pod writes verbose logs into an emptyDir and the node's disk fills. What happens, in order, and what stops it?

What happens, step by step

1. The kubelet detects disk pressure when the node filesystem crosses its eviction threshold - 10% available by default for nodefs. This is a node-level condition, not a pod-level one.
2. The node is tainted with disk pressure, so the scheduler stops placing new pods on it.
3. The image garbage collector runs, deleting unused container images to recover space. This succeeds and has a consequence later.
4. The kubelet starts evicting pods. Selection is by QoS class and by how far each pod exceeds its request for the resource under pressure. BestEffort pods go first, then Burstable ones ordered by overage.
5. Evicted pods are terminated mid-request. Callers see connection resets and 502s from services that have nothing to do with logging.
6. Rescheduled pods pull their images again, because step 3 deleted them - so the recovery produces a burst of image pulls that can itself be rate-limited by the registry.

The blast radius of one noisy pod is the whole node, and the services affected are chosen by QoS class rather than by culpability.

What the user sees

Intermittent errors across several unrelated features for a few minutes, then recovery, then a repeat the next time the log-chatty service is busy. In a dispatch system that means failed assignment calls during a meal peak - the worst possible timing, because the noisy pod is noisy precisely when traffic is high.

What stops it

• Declare ephemeral-storage requests and limits per container. With a limit set, the kubelet evicts the pod that exceeded its own limit, not a lottery of neighbours. This single setting converts a node-wide incident into one pod restarting.
• Do not write logs to a volume at all. Write to stdout and let the container runtime handle rotation with a configured size cap, which is bounded by design.
• Separate the runtime's filesystem from the node's, so image and container storage cannot be consumed by application scratch space.
• Monitor nodefs.available and per-pod ephemeral usage, and alert before the threshold rather than on the eviction events.

What would have to be true for it to self-heal

Only that the writer stops. Eviction recovers space and the node recovers, which is exactly why this recurs - the system heals itself and no one investigates. The signal to look for is a repeated eviction count on the same node, not a sustained alert.

When this is not worth defending against

On a single-tenant node pool running one application, the blast radius is already that application, and ephemeral-storage limits add configuration for little gain - though the log rotation is worth having anyway. The defence matters in proportion to how many unrelated workloads share the node, which is the general rule for every noisy-neighbour control.