A worker's queue depth grows steadily through the day and never recovers. Adding workers helps for an hour, then it resumes. What is happening?

The diagnosis

Arrival rate exceeds service rate. A queue that grows monotonically is not a queueing problem, it is a capacity problem, and no amount of buffering fixes it — the queue is simply recording the deficit.

Adding workers helping "for an hour" is the signature: the extra capacity drains the accumulated backlog, and once drained the underlying imbalance reasserts itself. That tells you the added capacity was enough to clear the backlog but not enough to exceed the arrival rate.

What to establish, in order

1. Measure both rates. Messages in per second and messages out per second, plotted together. If in > out at steady state, everything else is secondary.

2. Check whether the consumer is the bottleneck or something downstream. A worker waiting on a database is not consuming; adding workers then adds contention rather than throughput, which often makes it worse.

3. Look for a per-message cost that has grown. A steady arrival rate with a falling service rate means processing got more expensive — a table that outgrew an index, a payload that got larger, a dependency that slowed.

4. Check for poison messages cycling through retries and consuming capacity repeatedly without ever succeeding.

Why the unbounded queue is itself the problem

The queue hid the imbalance until it was severe. With a bounded queue, the producer would have been rejected as soon as capacity was exceeded — a loud, immediate, attributable failure rather than a slow accumulation discovered when latency became unacceptable.

Bounding also forces the useful question at design time: when this fills, what do we want to happen? Block the producer, reject, drop the oldest, or drop by priority — four different business decisions that an unbounded queue defers indefinitely.

The fixes

Right the capacity imbalance (more workers, cheaper processing, or less work). Bound the queue so the next imbalance is visible immediately. Add backpressure to the producer so it slows rather than accumulating. Monitor queue depth and age of oldest message — age is the better alert, because it maps directly to user-visible staleness.

What a strong answer adds

Noting that queue depth alone is a poor alert: a deep queue draining fast is fine, and a shallow queue that is not moving is not. Alert on oldest-message age and on the in/out rate ratio.