Per-Dependency Concurrency Limit

A service calling many downstreams from a shared pool has a shared fate: whichever downstream is slowest consumes the most resources, and at sufficient slowness it consumes all of them. The service then fails for every caller, including those whose dependencies are perfectly healthy.

A per-dependency limit — a semaphore, or a separate connection and thread pool — caps in-flight calls to each downstream. However slow one becomes, it can occupy only its allocation.

Why it matters

Slow is worse than down. A dependency that fails fast returns your resources immediately. One that takes eight seconds to eventually succeed holds a connection and a thread for eight seconds per call, and an error-rate circuit breaker sees nothing wrong because every call succeeds.

The limit is more reliable than a breaker because it is a hard bound rather than a reaction: no threshold to tune, no detection window during which damage accumulates, and no dependence on correctly identifying that something is wrong.

Implementation patterns

• Size the limit from throughput and acceptable latency, not from the downstream's capacity. If you can tolerate 2 seconds and it normally takes 200ms, a small limit suffices for large throughput.
• Prefer separate pools when the isolation must be absolute, and a shared pool with per-dependency semaphores when the overhead of many pools is not justified.
• Combine with a deadline derived from the caller's remaining budget, so a call that cannot complete in time is never started.
• Add a circuit breaker as the secondary control, tripping on latency as well as error rate, to stop calling a dependency entirely and give it room to recover.
• Make rejection fast and explicit when the limit is reached, so the caller can fall back rather than queue.
• Validate with fault injection. Limits and breakers that have only been tested against mocks have not been tested against saturation.

Industry example

A financial-data aggregator such as Plaid integrates hundreds of external banking APIs with wildly different latency profiles and reliability. A single shared worker pool means one bank's slow afternoon becomes a platform-wide outage. The per-provider limit is what converts that into a per-provider degradation, and it is the difference between an incident affecting one institution's users and one affecting everybody.

The same structure applies to any platform fanning out to many third parties: shipping aggregators over carriers, insurance platforms over underwriters, payment orchestrators over acquirers.

Failure scenarios

• One shared pool for all downstreams, giving the slowest one control of the whole service.
• Limits set per instance rather than per cluster, so the aggregate against a fragile downstream is far larger than intended.
• Rejection that queues instead of failing fast, which recreates the exhaustion the limit was meant to prevent.
• No fallback behaviour on rejection, so the limit converts a slow response into an error rather than into a degraded but useful one.
• Untested thresholds, which are guesses until an injection exercise shows otherwise.

Trade-offs

A limit set too low rejects requests a healthy dependency could have served, converting available capacity into errors. Set too high, it fails to contain the exhaustion it exists to prevent.

The resolution is usually adaptive limits — a controller that raises the ceiling while latency is good and lowers it as latency rises — which removes most of the tuning burden at the cost of a more complex mechanism that itself needs to be understood when it misbehaves.

The deeper trade is that isolation costs utilisation: reserving capacity per dependency means some of it sits idle. That is the price of not having a single slow downstream take everything, and it is almost always worth paying.

Interview question

"One of your two hundred provider integrations starts taking twelve seconds to respond, still returning 200s. Describe what happens to your service with a shared pool, and what changes with per-provider limits — including what the user of a healthy provider experiences in each case."