A platform must size worker pools for a peak of 50,000 requests per second with a 200ms average service time. How does Little's Law inform the answer, and what does it not tell you?

What the law gives

Concurrency = arrival rate × service time. At 50,000 requests per second and 200ms each, the system must sustain 10,000 concurrent requests in flight. That number is the input to every downstream sizing decision: thread counts, connection pools, memory per request, and the number of instances.

It also works in reverse and this is the more useful direction in practice: if you can only sustain 2,000 concurrent requests, then at 200ms service time your maximum throughput is 10,000 per second, regardless of how many instances you add — because the constraint is the concurrency limit somewhere in the path, not the compute.

What it does not tell you

• That service time is constant. It is not. Service time rises with load — cache hit rates fall, contention increases, garbage collection grows. Applying the law with a service time measured at low load under-provisions, sometimes badly, because the real peak service time may be double.
• Where the constraint is. The law says you need 10,000 concurrent; it does not say whether the limiting resource is CPU, connections, memory, a downstream service or a per-account lock.
• What happens beyond the limit. The law describes a stable system. Beyond capacity, queue length grows without bound and latency grows with it, which is a different regime entirely and the one that matters during an incident.
• The variability. Two systems with identical averages behave completely differently if one has consistent service times and the other has a long tail. Queueing behaviour is driven by variance, not by the mean, and this is where the simple calculation most often misleads.

How to use it properly

Measure service time at peak-like load, not at rest. Then compute required concurrency, then check every resource on the path against that number — connection pools, thread pools, downstream limits, memory. The peak finds whichever ceiling is lowest, and enumerating them is the exercise that turns a calculation into a capacity plan.

And add headroom for three separate things: forecast error, failure tolerance (capacity after losing an instance, not before), and the degradation margin from service time rising under load.