A service becomes unresponsive during an incident in a dependency it barely uses. Investigation finds the client had no read timeout. Explain the full mechanism.
Show the full answer Hide the answer
The mechanism, step by step
- The dependency slows. It has not failed — it accepts connections and eventually responds, just very slowly. This is the important detail: a hard failure would have been survivable.
- The client has no read timeout, which is the default in several popular HTTP clients. Each call waits indefinitely.
- Each waiting call holds a connection and a thread from a pool shared across all dependencies.
- Little's Law does the damage. In-flight work equals arrival rate times latency. Even at a low call rate — this dependency is "barely used" — a latency increase from 50 ms to indefinite means occupancy grows without bound. Every call ever made to it is still in flight.
- The shared pool exhausts. Now requests that never touch the slow dependency cannot obtain a connection either.
- Health checks share the pool, so they start failing, and the platform begins restarting healthy instances — removing capacity from a service that is already unable to serve.
- Callers time out and retry, multiplying the load on the remaining instances.
The initiating fault was one slow dependency used for a fraction of traffic. Everything after step 3 was the system attacking itself.
The fixes, in priority order
A read timeout derived from the dependency's p99. This alone bounds occupancy. It is the smallest possible change and it removes the unbounded term.
A bulkhead — a separate pool per dependency. Then exhaustion is contained to the one code path, and the "barely used" dependency can only consume its own small pool. This is the more fundamental fix and it is what makes step 5 impossible.
Health checks that do not share the pool, and liveness probes that test the process rather than its dependencies — otherwise the platform amplifies the outage.
A circuit breaker with a fallback, so once the dependency is clearly failing you stop calling it at all and serve something useful.
What a strong answer adds
Noting that "barely used" made it more dangerous, not less: nobody had load-tested that path, its latency was not on any dashboard, and no alert covered it. The dependencies that hurt you are usually the ones nobody thought were important enough to instrument.