Retry Budget

Per-request retry policies are locally reasonable — three attempts with backoff is sensible for one call — and globally catastrophic, because when a dependency degrades, every request retries and load on the struggling dependency multiplies by the retry count.

A retry budget inverts the control: retries may consume no more than some fraction of outbound requests (commonly around 10%) in a rolling window. When the budget is exhausted, calls fail fast without retrying.

Why it matters

A retry storm is self-sustaining. The dependency cannot recover while receiving several times its normal load, and it will keep receiving it because the retries are automatic and each one is individually justified. The budget is the only control that bounds the aggregate, because it is the only one expressed in aggregate terms.

Implementation patterns

• Express the budget as a ratio, not a rate, so it scales automatically with traffic instead of needing to be retuned.
• Keep the budget per dependency, so one struggling downstream does not consume the retry allowance of a healthy one.
• Combine with exponential backoff and full jitter. Backoff reduces frequency; jitter desynchronises clients, and without it a storm simply becomes a series of thundering herds.
• Make it adjustable at runtime. The moment you need to stop retrying is during an incident, and a change that requires a deploy is not a mitigation.
• Retry at one layer only — the edge closest to the user, which is the only layer that knows whether anyone is still waiting. Retries stack multiplicatively: three layers of three attempts is twenty-seven calls from one user action.
• Retry only what is safe. A timeout on a non-idempotent write is ambiguous, and the correct default for anything moving money is not to retry but to reconcile.

Industry example

Wallet and payment platforms such as MobiKwik sit between bursty consumer traffic and bank endpoints with limited and unpredictable capacity. The characteristic incident is not a bank outage but a partial degradation that the platform's own retries convert into a total one: success falls to 50%, retries quadruple the load, and the bank stops responding altogether.

The diagnostic that identifies it: load on a dependency rising while its success rate falls is a retry storm, not a traffic increase.

Failure scenarios

• Budget exhausted by a genuinely transient blip, causing fast failures that a retry would have absorbed. This is the intended trade and it should be understood, not treated as a bug.
• Budget configured globally instead of per dependency, letting one bad downstream starve retries for all.
• Backoff without jitter, producing synchronised herds.
• Retries at multiple layers, none of which knows about the others.
• A budget that cannot be changed without a deploy, and therefore is not available during the incident it exists for.

Trade-offs

A budget deliberately makes some requests fail that a retry would have rescued. That is the point: it trades a small number of avoidable failures for the survival of the dependency, and in a shared system the second is worth far more.

The tuning question is the ratio. Too tight and normal transient errors surface to users; too loose and the storm still forms. Ten percent is a common starting point and the correct value is whatever leaves the dependency room to recover under its observed failure modes — which means it should be validated by fault injection rather than chosen from a blog post.

Interview question

"Your service retries three times with exponential backoff, which is textbook. A downstream dependency drops to 50% success and your service takes it down completely. Explain the mechanism, and tell me what you change — including what you would change during the incident itself."