Why is limiting concurrency more effective than limiting request rate when protecting a service?

What is being tested

Whether you understand which quantity actually exhausts a system, and the relationship expressed by Little's Law.

The reasoning

L = λW. Concurrency equals arrival rate times latency.

A rate limiter bounds λ. But resources are consumed by L — each in-flight request holds a thread, a connection, memory and a socket for its entire duration.

So consider a service happily handling 500 requests per second at 200 ms, giving concurrency of 100. A downstream slows to 3 seconds. The arrival rate has not changed at all, so the rate limiter is perfectly content. But concurrency is now 1,500. The thread pool, sized for 100 plus headroom, is exhausted within seconds.

The rate limiter did exactly what it was configured to do and provided no protection whatsoever against the failure that actually occurred.

A concurrency limit of, say, 150 would have capped in-flight work regardless of latency: excess requests queue briefly and are then rejected quickly, the service stays responsive for what it accepted, and the failure is bounded and visible instead of total.

What concurrency limiting gives you

• Protection that adapts automatically to downstream latency. No configuration change needed when a dependency slows.
• A natural expression of what the resource can absorb, since pools are concurrency limits already — the limit just makes it explicit and adds a shedding policy instead of an exhaustion failure.
• Fast rejection instead of slow collapse. A rejected request costs almost nothing; a request that hangs for 30 seconds and then times out has consumed a slot for 30 seconds.

Where rate limiting is still correct

They solve different problems and both are needed:

• Rate limiting enforces fairness and business policy — plan quotas, per-tenant entitlements, abuse control. It is about who may use how much.
• Concurrency limiting protects capacity. It is about how much work can be in flight.

A complete design has rate limiting at the edge for policy and concurrency limiting in the service for protection.

The refinement worth knowing

Adaptive concurrency limits, which adjust the limit based on observed latency in the same way TCP congestion control adjusts its window. A static limit is correct at one point in the system's life and wrong afterwards — as instance sizes, dependencies and code change. Adaptive limits track the actual capacity rather than a number someone chose two years ago.

The prerequisite

Bulkheads: a separate limit per dependency. A single shared pool means one slow dependency exhausts capacity for all of them, which is the failure this is meant to prevent.