A brief database slowdown caused a two-hour full outage. Explain the likely amplification chain and the fixes at each stage.

The chain

Slow database → queries that took 10 ms take 2 s.

Thread pool exhaustion. Requests hold threads for 200× longer. A pool of 50 that handled the load easily now saturates, and requests that touch nothing related to the database also queue.

Client timeouts. Callers time out, and retry — adding load to an already-saturated service.

Retry multiplication. Retries configured at three layers — client library, gateway, service mesh — multiply. Three retries at each of three layers is up to 27 attempts per original request. The system is now attacking itself, and this is usually the largest amplifier.

Health check failure. Saturated instances fail health checks and are removed from rotation, concentrating load on the survivors, which then fail. Or worse, a database check in the liveness probe restarts every instance simultaneously, adding cold-start cost to an already failing system.

Recovery prevention. When the database recovers, the accumulated retry backlog hits at once and knocks it over again. This is why the outage lasted two hours rather than two minutes.

Fixes, stage by stage

Timeouts proportionate to normal latency, decreasing inward, so a slow dependency cannot consume the whole request budget.

Retry budgets, not retry counts. Cap retries at a small percentage of total requests — retries stop automatically when the failure is systemic, which is exactly when they are harmful. Add jitter, and retry at one layer only, chosen deliberately.

Circuit breakers so a failing dependency stops receiving traffic and gets room to recover.

Concurrency limits converting exhaustion into fast rejection, with bulkheads per dependency so one slow dependency cannot consume the shared pool.

Load shedding by priority at the edge, so partial capacity serves the most important traffic rather than failing everything equally.

Correct health check semantics — dependency checks in readiness, never in liveness.

Backoff on recovery — drain the backlog gradually rather than releasing it in one burst.

The postmortem framing

There is no root cause here. The database blip was the trigger; the outage was produced by timeouts, retry configuration, pool sizing, health check semantics and the absence of shedding. Each is a separate fix and each generalises to incidents you have not had yet.

Stopping at "the database was slow" produces one action item and leaves the amplifier intact.

What a strong answer adds

Noting that every amplifier here is resilience machinery behaving as configured. Retries, health checks and autoscaling are all meant to help, and all made it worse. That is why resilience configuration needs testing — failure injection in the service's own test suite, so a retry-storm regression fails the build rather than the platform.