A service's latency rises sharply while throughput plateaus and CPU sits at 50%. Use queueing theory to explain what is happening and what to measure next.

What the symptoms mean together

Throughput plateauing while latency rises is the signature of a saturated resource that is not the CPU. Throughput has hit a ceiling, so additional load cannot be served faster and instead accumulates as queue depth, which appears as latency.

CPU at 50% eliminates compute as the bottleneck and points at something with a fixed concurrency limit: a connection pool, a thread pool, a lock, a downstream dependency, or an I/O device.

The arithmetic

Little's law: L = λW. Concurrency equals arrival rate times residence time.

Rearranged for capacity: maximum throughput = concurrency limit ÷ service time. A connection pool of 50 against a database whose queries take 25 ms yields a hard ceiling of 2,000 requests per second, regardless of how many CPUs the machine has. Requests beyond that wait for a connection, and that wait is pure added latency with no work being done.

This is why the CPU is idle: the threads are blocked, not busy.

Why the degradation is non-linear

Queueing theory gives the shape: as utilisation ρ approaches 1, queue length grows as roughly ρ/(1−ρ). At 50% utilisation the queue is about 1. At 90% it is 9. At 99% it is 99.

The system does not degrade gradually — it is fine, then it is fine, then it collapses. This is why capacity planning based on average utilisation is dangerous: the useful headroom is far smaller than the arithmetic suggests, and the last 10% of utilisation contains most of the latency.

Variability makes it worse. Higher variance in arrivals or service times produces longer queues at the same utilisation, which is why bursty traffic needs more headroom than smooth traffic at identical averages.

What to measure next

1. Concurrency and queue depth per resource — connection pools, thread pools, semaphores. The saturated one is the answer, and this is the measurement most systems lack.
2. Wait time versus service time, separately. A request spending 200 ms waiting for a connection and 25 ms executing tells the whole story, and a single latency number tells none of it.
3. Utilisation of each pool, and how close to 1 it runs at peak.
4. Downstream latency and concurrency, since a dependency that has slowed increases residence time and therefore the concurrency needed for the same throughput.
5. Lock contention and I/O wait, for the cases where the limit is not an explicit pool.

The fixes, in order

• Reduce service time, which raises the ceiling proportionally and is usually the best fix: a faster query, an index, a cache. Halving service time doubles maximum throughput at the same concurrency limit.
• Raise the concurrency limit, but only if the downstream resource can absorb it — enlarging a connection pool in front of an overloaded database moves the queue rather than removing it, and typically makes things worse by increasing contention.
• Reduce arrival rate through caching, batching or shedding.
• Add adaptive concurrency limits, so the service sheds rather than queues when residence time rises.
• Bound every queue, so saturation produces fast rejection rather than unbounded latency. An unbounded queue converts a capacity problem into a timeout problem for every user.