A queue-backed worker pool processes image jobs. One consumer's process is alive and its health check passes, but it is blocked in a socket read to a storage service that never returns and has no timeout. Queue depth looks normal. What happens over the next hour and what would have caught it?

Minute by minute

Minute 0. The consumer has pulled a batch of messages. The broker has made them invisible to other consumers for the length of the visibility timeout or the lease, so they are neither being worked on nor available to anyone else.

Minutes 1 to 5. The lease expires. The broker redelivers those messages to a healthy consumer, which processes them. If the work is not idempotent, the job has now run twice. The zombie will never acknowledge, so every message it touches is delivered at least twice for as long as it lives.

Minutes 5 to 60. The zombie keeps polling, because polling is a different code path from the one that is stuck, or it keeps holding the messages it took. Throughput has dropped by one consumer in N, which at ten consumers is a 10% capacity loss that nothing alerts on. Queue depth stays normal because the other consumers absorb the work, which is exactly why depth is the wrong signal.

The nastier variant: if the work is partitioned by key and the zombie owns a partition, that partition stops entirely. Global throughput barely moves. One customer's jobs stop completely.

Where it amplifies

Redelivery is not free. Each redelivered message costs the work already done plus the work done again. If the stuck dependency is shared, several consumers stick at once and redelivery multiplies load on the very service that is struggling. A slow dependency plus retries on lease expiry is a load amplifier, and it feeds the storage service more work precisely when it is failing.

What the user sees

Nothing, then duplicates. Two thumbnails, two charges, two emails. For a partitioned workload, one tenant sees their jobs stop while the status page is green.

What stops it

1. A deadline on every network call, propagated from the job's own budget. A socket read with no timeout is the root cause here; the lease is only the thing that made it visible.
2. Liveness that means progress, not process. The health check must assert that the consumer acknowledged a message within some window, not that the process can answer HTTP.
3. Alert on age of the oldest unacknowledged message and on per-consumer acknowledgement rate, not on queue depth. Depth is a measure of arrival versus service across the pool and hides a single failed member.
4. Idempotency keyed on the job, so redelivery is safe by design rather than by luck. This is required regardless: every at-least-once broker will redeliver eventually.

What would have to be true for it to self-heal

The runtime would have to detect the stuck thread and restart the process. Nothing in a normal service does that. Self-healing here is a timeout, or it does not exist - the orchestrator restarts a pod that fails its probe, so the probe is the only mechanism that converts a hang into a restart.

When this is not worth engineering for

A single-consumer batch job that runs nightly and is watched by a human has none of this risk profile. The pattern matters when a pool hides the failure of one member, which is the defining property of competing consumers.