Concurrency Limiting
Bounding the number of simultaneous in-flight operations so that overload produces fast rejection rather than resource exhaustion.
Rate limiting bounds requests per second. Concurrency limiting bounds requests in flight, and it is the more direct protection, because what exhausts a service is simultaneous work holding threads, connections and memory — not arrivals.
A service handling 100 requests per second at 10 ms each has one request in flight. The same arrival rate at 2 s each has 200 — and if the thread pool holds 50, the other 150 queue, latency climbs, clients time out and retry, and the queue grows faster. That is the classic collapse, and a rate limit set on requests per second does not prevent it.
With a concurrency limit, the 51st concurrent request is rejected immediately. Fast failure is a much better outcome than slow failure: the client can retry elsewhere or degrade, and the service stays responsive for the requests it accepted.
Applied per dependency, it is the implementation of a bulkhead — a slow dependency consumes its own budget and no more. Applied per tenant, it is fairness.
Adaptive limits — adjusting the ceiling from observed latency, as in gradient or AIMD algorithms — avoid the tuning problem, since a static limit is either too low at good times or too high at bad ones.