Queue Length Estimation
Using the relationship between arrival rate, residence time and items in the system to size pools, predict backlogs and sanity-check capacity claims.
The relationship — items in system = arrival rate × time in system — holds for any stable system regardless of distribution, which is what makes it useful as an arithmetic check rather than a model.
Three uses that come up constantly:
Sizing a connection or thread pool. 500 requests per second at 40 ms each means 500 × 0.04 = 20 concurrent requests. A pool of 20 is the minimum; a pool of 200 is wasting memory, and a pool of 5 is the bottleneck. Most pool sizes are guessed and this takes thirty seconds.
Predicting a backlog. A queue receiving 1,000 messages per second with consumers processing 800 per second accumulates 200 per second. Time to a given depth follows directly, and so does the number of consumers needed to drain it in a stated window.
Testing a claim. If a team says a service handles 10,000 requests per second with 100 ms latency, that implies 1,000 concurrent in-flight requests. If the thread pool is 50, the claim is wrong — either the latency or the throughput figure is not what was measured.
The caveat: it holds only for a stable system. If arrivals exceed the departure rate, there is no steady state and the queue grows without bound.