A downstream service slows from 50 ms to 3 s. Within two minutes every service in the request path is down, including ones that do not call it. Explain the mechanism and how you would have prevented it.

What the interviewer is testing

Whether you understand that most outages are amplification, not failure — and whether you can name the specific mechanism rather than saying "add a circuit breaker".

The mechanism, step by step

1. The slow dependency holds each caller's thread or connection for 3 seconds instead of 50 ms — a 60-fold increase in occupancy.
2. By Little's Law, in-flight requests are arrival rate times latency. At constant traffic, the caller's concurrency needs rise 60-fold. Its thread pool or connection pool exhausts.
3. Once the pool is exhausted, the caller cannot serve any request, including ones that never touch the slow dependency. This is why services that do not call it also fail: they share the pool.
4. Callers time out and retry. Retries multiply load on a service that is already saturated, so it gets slower, so more callers time out. The loop is now self-sustaining.
5. Retries at multiple layers multiply. Three layers each retrying three times is 27 requests for one user action.
6. Health checks share the pool, so the platform marks healthy instances unhealthy and restarts them, removing capacity from an already-saturated service.

The initiating fault was a slowdown. Everything after step 2 was the system attacking itself.

The prevention, in priority order

Bulkheads. Separate connection pools per dependency. This alone stops step 3: a slow dependency exhausts its own pool and nothing else. It is the single highest-value fix and the most frequently missing.

Timeouts derived from a budget, not from a guess. Give the request an overall deadline at the edge and propagate the remaining budget with each hop. A downstream call that cannot finish within the remaining budget should not be made at all.

Retry with a budget, not a count. Cap retries as a proportion of traffic — for example, retries may not exceed 10% of requests. This is what stops step 4, and it is strictly better than "three attempts", which multiplies under exactly the conditions where multiplication is fatal.

Exponential backoff with jitter. Without jitter, backoff still produces synchronised waves.

Circuit breakers. Once the failure rate crosses a threshold, stop calling and fail fast to a fallback. Note this is fourth on the list: it limits damage, it does not prevent the mechanism.

Retry at one layer only. Decide where retries live — usually the outermost caller that knows the user's deadline — and make every other layer fail fast.

Health checks that do not share the pool. A liveness probe should test the process, not the dependencies, or the platform will amplify the outage on your behalf.

What a strong answer adds

Load shedding: when saturated, reject the excess immediately with 429 rather than queueing it, and prefer to shed work that has already burned most of its deadline, since completing it helps nobody.