During a regional degradation, a mobility platform's services perform millions of retries against a slow dependency, and the dependency never recovers until traffic is manually cut. Analyse the failure and identify which four controls would have prevented it.

What happened, mechanically

The dependency slowed — not failed. Callers hit their timeouts and retried. Each retry added load to a system already at capacity, which slowed it further, which caused more timeouts, which caused more retries.

The important property is that this state is metastable: it persists after the original trigger is gone. Even if the initial cause was a transient blip, the system is now generating enough self-inflicted load to stay broken. That is why removing traffic manually was the only thing that worked — the system had no path back on its own.

The load multiplier is worse than it looks. With three retries per call and three layers of services each retrying, the amplification is 3³ = 27x, not 3x. Retry amplification compounds multiplicatively through the call graph.

The four controls

1. Retry budgets, not retry counts. A per-client budget — retries may not exceed some small percentage of successful requests over a rolling window — caps total retry load regardless of how many callers there are. Unlike a per-request count, it degrades gracefully: when everything is failing, almost nothing gets retried, which is exactly the correct behaviour.

2. Exponential backoff with full jitter. Backoff alone is insufficient — synchronised clients retry in synchronised waves. Full jitter (a random delay uniformly drawn up to the current backoff ceiling) spreads the load rather than moving the spike. This is one of the highest-value, lowest-effort changes in distributed systems.

3. Deadlines propagated through the call graph. A request carries the time remaining. If a caller's deadline has already expired, downstream work is wasted by construction — the response will be discarded. Deadline propagation makes the system stop doing work nobody is waiting for, which is a large fraction of the load during a cascade.

4. Circuit breakers with a real half-open probe. Once failure rate crosses a threshold, stop calling. Fail fast, use the fallback, and let a single probe request test recovery rather than the whole fleet testing it simultaneously. Without this, the moment the dependency starts recovering it is immediately re-buried.

The control that is missing from most lists

Retry only what is retryable. A 4xx-class error, a validation failure, or a deterministic downstream rejection will fail identically every time. Retrying non-retryable errors is pure amplification with zero chance of success, and it is extremely common because generic HTTP clients retry on anything that is not a 2xx.

The design lesson

Retries are a local optimisation with a global cost. Every individual retry is rational — it improves that request's chance of success. The aggregate is a positive feedback loop. This is why retry policy belongs to the platform, in a shared client or a service mesh, rather than being a decision each team makes independently.