Consumer Liveness Gap

A worker pool's dashboards are pool-level by design. Queue depth, throughput and error rate describe the group. A single consumer that stops progressing without dying is invisible in all three, because the others absorb its share and the graphs stay flat.

The usual cause is a network call with no deadline. A socket read to a storage service that never returns leaves the process running, its HTTP health endpoint answering, its metrics being scraped, and its actual work permanently stopped.

Why it matters

The failure is silent and it compounds. Each message the zombie holds is redelivered when its lease expires, so everything it touches is processed at least twice, and if the work is not idempotent the duplicates are real: two charges, two emails, two shipments.

Worse, when work is partitioned by key and the stuck consumer owns a partition, that partition stops entirely. Global throughput drops by one in N and nothing alerts, while one customer's work has completely halted behind a green status page.

Implementation patterns

• A deadline on every network call, derived from the job's own budget rather than set per call. This removes the cause rather than detecting the symptom.
• Liveness that asserts progress. The probe fails unless the consumer has acknowledged a message, or explicitly polled an empty queue, within a window - typically a small multiple of the expected processing time. The orchestrator then restarts the pod, which is the only mechanism that converts a hang into recovery.
• Alert on age of the oldest unacknowledged message and on per-consumer acknowledgement rate. Both reveal a single stalled member; depth does not.
• A lease long enough to cover p99 processing and no longer, because the lease is also the detection delay for a dead consumer.
• Idempotency keyed on the unit of work, required regardless, because every at-least-once broker redelivers eventually.
• Per-partition lag metrics where work is keyed, so a single stalled partition is visible rather than averaged away.

Industry example

The adjacent documented case is Discord's 2020 account of moving its Read States service from Go to Rust, where latency spikes came from garbage collection in the runtime rather than from load: the process was healthy by every ordinary measure and periodically stopped serving. The lesson transfers directly, and the general form is that aggregate health signals hide per-member and tail failures. A pool's dashboards are aggregates by construction, so the fix is a metric that cannot be averaged away: per-member acknowledgement rate, and the age of the oldest thing not yet done.

Failure scenarios

• A stuck consumer redelivering its whole batch every lease period, adding load to the dependency that is already failing.
• Duplicate side effects from redelivery of non-idempotent work.
• A frozen partition with normal aggregate throughput.
• A health check that passes because the HTTP server thread is independent of the stuck worker thread, which is the default in almost every framework.
• Autoscaling scaling down because throughput per consumer looks low, removing healthy consumers and leaving the zombie.

Trade-offs

Progress-based liveness has a false-positive cost: a consumer legitimately working on a long job, or sitting on an empty queue, can be restarted by a probe that is too aggressive. The window must account for the longest legitimate unit of work, which means the detection delay is bounded below by the slowest job you allow. Splitting long jobs into checkpointed steps narrows both.

Shorter leases detect failure faster and increase duplicate delivery for slow jobs. There is no setting that avoids both; there is only idempotency, which makes the choice safe.

When not to use it

A single-consumer job, or a batch process a human watches, has no pool to hide the failure, and the ordinary alert on job completion covers it. The gap is a property of pools: it appears exactly when redundancy makes one member's failure invisible, which is the same property that made the pool attractive.

Interview question

Q: One worker in a pool of ten is alive but making no progress. Queue depth is normal and no alert has fired. Tell me what you would have instrumented beforehand, and what you would change so that a restart happens without a human.

What a strong answer covers: why depth is a pool metric and cannot show a single stalled member · age of oldest unacknowledged message and per-consumer acknowledgement rate as the signals · a progress-asserting liveness probe as the automatic remedy · deadlines as the root-cause fix · lease length as both a detection delay and a duplication source · and idempotency as the precondition that makes redelivery safe.

Quick check

Quiz: Why does queue depth not reveal a stalled consumer? Because depth compares arrival against the whole pool's service rate, and the remaining consumers absorb the stalled member's share.

Flashcard: Which two signals reveal one stuck consumer? Age of the oldest unacknowledged message, and acknowledgement rate per consumer.