A telemetry ingestion queue grows continuously even though every worker reports healthy and CPU is moderate. How do you distinguish between insufficient capacity, a slow downstream dependency, oversized messages, poison messages and an incorrect workload model?

The first measurement, before any hypothesis

Compare arrival rate and service rate directly, and measure time in queue rather than queue depth. Depth is a symptom that conflates causes; the age of the oldest unprocessed message tells you whether you are behind by seconds or hours, and the ratio of arrival to service rate tells you whether you are diverging or merely lagging.

If arrival ≈ service and depth is flat but large, you have a latency problem. If arrival > service, you have a throughput problem, and everything below is about locating it.

Distinguishing the five causes

Insufficient capacity. Workers are busy — high utilisation, high CPU or high in-flight count — and adding workers increases throughput roughly linearly. The tell is that utilisation is high. In this scenario CPU is moderate, so this is probably not it, and adding workers will not help.

Slow downstream dependency. Workers are busy waiting, not computing. Look for high in-flight count with low CPU — exactly the reported symptom. Measure the time each worker spends blocked on downstream calls as a fraction of processing time. If it is dominant, the queue is a display of a bottleneck that lives elsewhere, and scaling workers will make the downstream problem worse.

Oversized messages. Throughput in messages per second falls while bytes per second stays flat or rises. Plot both. A customer that started sending payloads ten times larger changes the workload without changing the message count, and every dashboard measured in messages will look mysterious.

Poison messages. A message that fails, is retried, fails again, and consumes a worker slot forever. The tell is redelivery count and a repeated message identifier in error logs — and the queue growing while successful throughput stays roughly constant. Requires a dead-letter queue with a retry ceiling, and it is worth noting that without a DLQ this failure is invisible in aggregate metrics.

Incorrect workload model. Ordering or partitioning constraints mean the work cannot actually be parallelised. If one tenant's data must be processed in order and one tenant is 90% of volume, you have one effective consumer no matter how many workers exist. Look for skew: throughput per partition, not in aggregate.

Why "healthy" workers are misleading

A health check answers "is this process running", which is almost never the question. A worker blocked on a 6-second downstream call is healthy and useless. Health checks should assert progress — messages completed in the last interval — not liveness.

What to do regardless of cause

Bound the queue. An unbounded queue converts a throughput problem into a memory exhaustion problem and, worse, into an unbounded latency problem: telemetry arriving hours late is not merely delayed, it is worthless, and the system is spending money to deliver worthlessness.

The correct behaviour at the bound is explicit load shedding with a stated policy — for observability data specifically, shedding by sampling low-value telemetry while preserving errors and traces linked to them is far better than uniformly dropping, and vastly better than falling over.