The retry is the outage

A field guide to metastable failure: the class of outage where the system stays down after the thing that broke it has been repaired, because the retries, caches, health checks and autoscaling installed to make it reliable are now the load. Reconstructed from twelve published incidents at AWS, GitHub, Google, Meta, Slack, Spotify and IBM, two systems papers, and the source of the proxies and client libraries that ship the countermeasures. After reading it you can compute your own stack's retry amplification, measure the hidden capacity below which it self-heals, and choose between a token bucket, a retry budget and an adaptive concurrency limit with the condition that flips each one.

What you get out of it

• Retries multiply rather than add: five service hops at three attempts each is 243 times the intended load, and nobody configures that number deliberately.
• Backoff with jitter is a scheduling policy, not a bound. It changes when the amplified work arrives, not how much of it there is; only a budget or token bucket caps the total.
• Retry policy is the sustaining effect in more than half of the 22 metastable failures studied at OSDI '22, and load shedding is the mitigation in more than half. Adding capacity mid-incident consumes capacity before it adds any.
• The recommended countermeasure has a known accounting gap: Envoy's retry budget counts retries in backoff against the limit but not in the denominator that sets it, the issue was closed as not planned in 2023, and three pull requests to fix or generalise it closed unmerged between 2024 and May 2026.
• Post-incident action items that increase persistence, retry count, logging detail or health-check aggressiveness strengthen the feedback loop. Ask of each one what the new worst-case work amplification is and where the cap sits.

Scope

Why this, now. GitHub lost seven hours and forty-seven minutes to client-side retry amplification on 17 August 2026, and its CTO's remediation list three days later is the same prescription Google published in 2016 and AWS acted on in 2015.

What it does not cover. Idempotency and exactly-once semantics, which decide whether a retry is safe rather than whether it is affordable; tracing and detection tooling; chaos engineering practice; the organisational half of incident response; and denial-of-service traffic, which resolves when the attacker stops and is therefore not metastable.