A wallet platform sees intermittent failures from a downstream bank, and its own retries turn a partial degradation into a total outage. What combination of controls prevents a retry storm?

How the storm forms

A dependency degrades to 50% success. Every failed call retries three times. Load on the dependency quadruples exactly as it is struggling, so success falls further, so more calls retry. The dependency is now receiving several times its normal traffic while serving almost none of it, and it will not recover until the retries stop — which they will not, because they are automatic.

Add synchronised backoff and every client retries at the same instant, producing a load pattern of periodic spikes that is worse than steady overload.

The controls, and what each one does

• Exponential backoff with full jitter. Backoff reduces the rate; jitter is what desynchronises clients and it is the half most often omitted. Backoff without jitter converts a storm into a series of thundering herds.
• A retry budget, expressed as a share of total requests — for example, retries may not exceed 10% of outbound calls in any window. This is the control that actually bounds the storm, because per-request retry counts are locally reasonable and globally catastrophic. When the budget is exhausted, calls fail fast.
• A circuit breaker per dependency, so a sustained failure stops the calls entirely rather than retrying each one. The breaker gives the dependency room to recover, which retries actively prevent.
• Retry only what is safe. A timeout on a non-idempotent write is ambiguous, and retrying it can duplicate a payment. Retry policy has to be per operation, not per client library, and the default for anything that moves money should be "do not retry, reconcile."
• Deadline propagation, so a retry that cannot complete before the caller gives up is never attempted. Work on a request nobody is waiting for is pure waste during overload.

The layer nobody counts

Retries stack multiplicatively across layers. The mobile client retries three times, the gateway retries three times, the service retries three times: one user action becomes twenty-seven calls to the bank. Every retry layer must be a deliberate decision with a documented owner, and the general rule is to retry at the edge closest to the user and nowhere else in the chain, because that is the only layer that knows whether anyone is still waiting.

The diagnostic

If load on a dependency rises while its success rate falls, you are looking at a retry storm and not a traffic increase. The fix in the moment is to stop retrying — which is why the breaker and the budget need to be adjustable at runtime rather than at deploy time.