A service calls four dependencies. How do you size its thread pools, and why is the total often smaller than people expect?
Show the full answer Hide the answer
The sizing rule
Little's Law, per dependency: concurrency = throughput to that dependency × its latency.
| Dependency | Calls/s | Latency | Concurrency | Pool |
|---|---|---|---|---|
| Auth | 500 | 10 ms | 5 | 8 |
| Catalogue | 500 | 40 ms | 20 | 30 |
| Pricing | 200 | 80 ms | 16 | 24 |
| Fraud | 50 | 200 ms | 10 | 16 |
Pools are sized above the computed concurrency to absorb variance and the tail, not by an order of magnitude.
Total: 78 threads. Most teams would have configured 200 shared, or four pools of 50.
Why the total is smaller than expected
Because intuition anchors on peak throughput rather than on concurrency, and the two differ by the latency factor. 500 requests per second sounds like it needs a lot of threads; at 10 ms each it needs five.
The second reason: people size for the worst imaginable latency rather than the observed distribution, which inflates every pool simultaneously.
Why over-sizing is actively harmful
It admits more concurrency into the dependency than it can profitably serve. More concurrent queries means more lock contention, more context switching and more I/O queueing — so per-call latency rises and total throughput can fall. The pool is doing useful work as a bounded queue, protecting the dependency from itself.
It also defeats the bulkhead: pools large enough to collectively exhaust the process's threads or the dependency's connection limit are not isolating anything.
The checks that catch real outages
Pool × instance count against the dependency's connection limit. This is the arithmetic that catches the classic failure where an autoscaler doubles instances and the database starts refusing connections.
Behaviour on exhaustion. Reject immediately with a fallback — that is the point. Queueing behind an exhausted pool quietly undoes the pattern, and an unbounded acquisition wait recreates the original problem.
What a strong answer adds
Distinguishing thread pool isolation from semaphore isolation: semaphores are far cheaper and give no protection against a call that blocks the calling thread forever, since there is no separate thread to abandon. Use semaphores for calls you trust to time out, thread pools for calls that can hang.