A downstream call goes from 50 ms to 3 seconds at constant traffic. Use Little's Law to explain what happens to the caller and why services that never call it also fail.
Show the full answer Hide the answer
What is being tested
Whether you can state the mechanism of cascading failure quantitatively rather than saying "add a circuit breaker".
The arithmetic
L = λW. At constant arrival rate λ, latency W increasing 60-fold increases concurrency L 60-fold.
Suppose the caller handles 200 requests per second to this dependency at 50 ms: concurrency is 10, and a pool of 25 is comfortable. At 3 seconds, required concurrency is 600. The pool of 25 is exhausted in well under a second.
Why unrelated services fail
Because the pool is shared. Once the caller's thread pool or connection pool is fully occupied by requests waiting on the slow dependency, the caller cannot process any request — including requests for endpoints that never touch that dependency.
The caller is now failing entirely, and its own callers experience it as a slow dependency. Their concurrency rises. Their pools exhaust. The failure propagates up the call graph, and services three hops away with no relationship to the original fault go down.
This is why an incident's blast radius is so often wildly larger than its cause.
The amplification that follows
- Callers time out and retry, multiplying load on a dependency that is already saturated.
- Retries at multiple layers multiply: three layers each retrying three times is 27 requests for one user action.
- Health checks share the pool, so instances fail their probes and the platform restarts healthy instances, removing capacity from an already-saturated system.
The initiating fault was a slowdown. Everything after step one is the system attacking itself.
The prevention, in priority order
1. Bulkheads. A separate connection and thread pool per dependency. This alone stops the propagation: the slow dependency exhausts its own pool and nothing else. Highest-value mitigation and the most frequently missing.
2. Timeouts derived from a budget. Set an overall deadline at the edge and propagate the remaining budget with each hop. A call that cannot finish within the remaining budget should not be made.
3. Retry budgets, not retry counts. Cap retries as a proportion of traffic — for example no more than 10% of requests may be retries. Strictly better than "three attempts", which multiplies under exactly the conditions where multiplication is fatal.
4. Exponential backoff with jitter, or backoff still produces synchronised waves.
5. Circuit breakers, which limit damage rather than preventing the mechanism — note they are fourth, not first. And their threshold must be reachable before the pool exhausts, which requires checking against these numbers rather than choosing a round figure.
6. Concurrency limits, which cap L directly regardless of W.
7. Health checks that do not share the pool, and that do not test dependencies.
What a strong answer adds
The non-linearity underneath: queueing delay rises sharply as utilisation approaches saturation — roughly 4x the service time at 80% utilisation, 9x at 90%. So a system running at 60% degrades gracefully under this and one running at 90% falls off a cliff. Headroom is a resilience feature, not waste.