An API platform has an average of 400 requests in flight and processes 2,000 requests per second. What is the average latency, and how is Little's Law used for capacity decisions?
Show the full answer Hide the answer
The relation
L = λW: concurrency equals arrival rate times latency. Here W = 400 / 2000 = 0.2 seconds.
Its power is that it holds for any stable system regardless of the arrival distribution, the service time distribution or the scheduling discipline. Knowing any two quantities gives the third, without any modelling.
How it is used
1. Sizing thread and connection pools. To handle 2,000 requests per second at 200 ms latency, you need 400 concurrent slots. A pool of 100 caps throughput at 500 per second regardless of available CPU — and this is the most common cause of a system that will not go faster despite idle hardware.
2. Predicting the effect of a latency change. If a dependency slows from 200 ms to 600 ms, required concurrency triples to 1,200 for the same throughput. If the pool is fixed at 400, throughput falls to about 667 per second. This is why a slow dependency causes a throughput collapse rather than merely a latency increase, and it explains the shape of many incidents.
3. Sizing downstream connection pools. A service handling 2,000 requests per second, each making one database call taking 20 ms, needs 40 database connections. Provisioning 200 wastes database resources; provisioning 10 caps throughput at 500 per second.
4. Detecting queueing. If measured concurrency exceeds what the relation predicts from arrival rate and service time, the difference is queue time. That decomposition is what separates "the work is slow" from "the work is waiting".
The capacity planning use
Little's Law converts a latency SLO into a concurrency requirement, and a concurrency requirement into a resource requirement. That chain is what makes capacity planning arithmetic rather than guesswork.
The failure it explains
A system that will not scale despite low CPU. The bound is a fixed concurrency limit — a pool, a semaphore, a worker count — and adding CPU changes nothing. Applying the relation identifies it immediately: measured throughput times measured latency equals the concurrency ceiling exactly, which is the signature of a hard limit rather than a resource constraint.