A streaming consumer's lag is growing steadily while workers show low CPU and no errors. What are the candidate causes and how do you distinguish them?
Show the full answer Hide the answer
The candidates and their distinguishing signals
| Cause | Signal |
|---|---|
| Slow downstream | Workers blocked in I/O wait; time concentrated in downstream calls |
| Partition skew | One partition's lag growing while others are flat |
| Poison message | Repeated redelivery of one offset; lag stuck rather than growing |
| Too few partitions | Lag growing uniformly; consumer count already equals partition count |
| Rebalancing loop | Frequent partition reassignment; consumption pauses repeatedly |
| Oversized messages | Byte throughput flat while message throughput falls |
| Inefficient processing | Per-message CPU far above what the work justifies |
Low CPU with growing lag points strongly at the first, because a CPU-bound consumer would show CPU. The workers are waiting for something.
The check order
- Per-partition lag, which immediately separates skew from a uniform shortfall. Uniform lag with idle workers means a downstream bottleneck; single-partition lag means a hot key or a stuck message.
- Where the time goes inside the consumer — a profile or a span breakdown. This distinguishes downstream waiting from processing inefficiency in one observation.
- Rebalance frequency, which is often the surprise answer: a consumer whose processing occasionally exceeds the session timeout is evicted, triggering a rebalance, during which nothing progresses — and the recovery makes the next timeout more likely. This is a feedback loop that looks like slow processing.
- Consumer count against partition count, since consumers beyond the partition count are idle and adding more changes nothing.
The fix that is usually wrong
Adding consumers. It helps only when consumers are CPU-bound and partitions are available. Against a slow downstream it deepens that downstream's queue and can tip it into failure; beyond the partition count it does nothing at all.
The structural question underneath
If lag grows during every peak and recovers afterwards, the consumer is sized for average rather than peak, and the choice is between capacity for the peak or accepting bounded lag as a designed behaviour. Bounded, recovering lag is a legitimate design — provided the recovery time is understood and the downstream consumers of that data know their freshness guarantee.
If lag grows and does not recover, throughput is genuinely insufficient and no amount of tuning changes that.