Evidence ledger
30 sources
Checked 29 Aug 2026
Evidence ledger
One row per claim in The retry is the outage: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: why a recoverable overload becomes a self-sustaining outage, and which overload-control mechanisms actually break the loop.
All links fetched 2026-08-29. One row per claim. Quotes are copied, not paraphrased.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Penn State / UNH / Rockset (ex-Meta) | Metastable Failures in Distributed Systems (HotOS '21) | paper | 2021-05 | 2026-08-29 | https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf | The root cause of this failure class is the sustaining feedback loop, not the trigger | "We consider the root cause of a metastable failure to be the sustaining feedback loop, rather than the trigger. There are many triggers that can lead to the same failure state" |
| 2 | same | same | paper | 2021-05 | 2026-08-29 | https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf | Systems deliberately run in the vulnerable state because it is more efficient | "many production systems choose to run in the vulnerable state all the time because it has much higher efficiency than the stable state" |
| 3 | same | same | paper | 2021-05 | 2026-08-29 | https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf | Hidden capacity is half of advertised capacity in the worked retry example | "Once the load exceeds 150 QPS, however, the system enters a vulnerable state"; database max is 300 QPS; "Recovery from the metastable state requires reducing the web application load to under 150 QPS or limiting retries to less than 20 QPS." |
| 4 | same | same | paper | 2021-05 | 2026-08-29 | https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf | Losing a 90% hit-rate look-aside cache is a 10x query amplification, and the look-aside shape prevents the priority fix | "In effect, losing a cache with a 90% hit-rate causes a 10x query amplification"; "This prioritization is unenforceable with a look-aside cache but trivial with a read-through cache." |
| 5 | same | same | paper | 2021-05 | 2026-08-29 | https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf | A priority system does not prevent metastable failure and can encourage amplification | "a geo-distributed system that added additional retries and failover destinations to improve its steady-state reliability resulted in a worst-case work amplification of over 100x" |
| 6 | same | same | paper | 2021-05 | 2026-08-29 | https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf | Elastic capacity is not a reliable recovery lever | "unless a stateful system is specifically designed to provide zero-impact elasticity, reconfiguration will reduce capacity in the short term" |
| 7 | Penn State / UNH / Twitter | Metastable Failures in the Wild (OSDI '22) | paper | 2022-07 | 2026-08-29 | https://www.usenix.org/system/files/osdi22-huang-lexiang.pdf | 22 metastable failures across 11 organisations; at least 4 of 15 major AWS outages of the decade | "we present an in-depth study of 22 metastable failures from 11 different organizations"; "at least 4 out of 15 major outages in the last decade at Amazon Web Services were caused by metastable failures" |
| 8 | same | same | paper | 2022-07 | 2026-08-29 | https://www.usenix.org/system/files/osdi22-huang-lexiang.pdf | Retry policy is the single most common sustaining effect | "By far, the most common sustaining effect is due to the retry policy, affecting more than 50% of the studied incidents" |
| 9 | same | same | paper | 2022-07 | 2026-08-29 | https://www.usenix.org/system/files/osdi22-huang-lexiang.pdf | Duration range and the modal outage length | "we have observed outages in a range of 1.5 to 73.53 hours, with 4 to 10 hours of outages being the most common (35% of incidents reporting the outage period)" |
| 10 | same | same | paper | 2022-07 | 2026-08-29 | https://www.usenix.org/system/files/osdi22-huang-lexiang.pdf | Load shedding was the most used mitigation, and teams could not size it | "Load shedding was the most popular mitigation effort used in over 50% of the incidents… without a proper understanding of the metastability and feedback loops, it is hard to know just how much the load needs to be reduced." |
| 11 | same | same | paper | 2022-07 | 2026-08-29 | https://www.usenix.org/system/files/osdi22-huang-lexiang.pdf | Fix-to-break: the AWS SimpleDB remediation set up the DynamoDB outage | "engineers decided that servers must continue to retry the locking service instead of giving up… A similar incident (AWS3) happened to the DynamoDB database about a year later. The storage nodes did not back out of retrying to get updated membership data" |
| 12 | same | same | paper | 2022-07 | 2026-08-29 | https://www.usenix.org/system/files/osdi22-huang-lexiang.pdf | Fix-to-break, second instance: diagnostic logging on the error path | "In SPF2, the additional logging after a load spike and initial retries increased the cost of each retry, adding more load to the system and causing more requests to retry." |
| 13 | same | same | paper | 2022-07 | 2026-08-29 | https://www.usenix.org/system/files/osdi22-huang-lexiang.pdf | Trigger mix is dominated by engineer action, not load | "Around 45% of observed triggers in Table 1 are due to engineer errors"; "Load spikes are another prominent trigger category, with around 35% of incidents reporting it" |
| 14 | same | same | paper | 2022-07 | 2026-08-29 | https://www.usenix.org/system/files/osdi22-huang-lexiang.pdf | A retry cap is a cap on the stable region | "a policy with at most two retries will not amplify the work more than three times, while the policy with no cap effectively leaves the system with no stable region" |
| 15 | AWS | Summary of the Amazon DynamoDB Service Disruption, Sept 20 2015 | postmortem | 2015-09 | 2026-08-29 | https://aws.amazon.com/message/5467D2/ | Retries held the metadata service down after the trigger had cleared | "Unavailable servers continued to retry requests for membership data, maintaining high load on the metadata service." |
| 16 | AWS | same | postmortem | 2015-09 | 2026-08-29 | https://aws.amazon.com/message/5467D2/ | The overload blocked its own remedy | "we were unable to add capacity to the metadata service because it was under such high load, preventing us from successfully making the requisite administrative requests" |
| 17 | AWS | same | postmortem | 2015-09 | 2026-08-29 | https://aws.amazon.com/message/5467D2/ | Recovery required pausing the source of the retries | at 5:06 AM PDT AWS "decided to pause requests to the metadata service. This action decreased retry activity, which relieved much of the load on the metadata service." Error rates had reached ~55% by 2:37 AM. |
| 18 | AWS | Summary of the Amazon Kinesis Event in Northern Virginia, Nov 25 2020 | postmortem | 2020-11 | 2026-08-29 | https://aws.amazon.com/message/11201/ | Restart is not a free recovery lever at fleet scale | "bringing front-end servers back online too quickly would create contention… we could only add servers at the rate of a few hundred per hour"; "It takes up to an hour for any existing front-end fleet member to learn of new participants." |
| 19 | AWS | Summary of the AWS Service Event in the Northern Virginia Region, Dec 7 2021 | postmortem | 2021-12 | 2026-08-29 | https://aws.amazon.com/message/12721/ | Retries turned a connection surge into persistent congestion, and the backoff that was supposed to stop it did not fire | "These delays increased latency and errors… resulting in even more connection attempts and retries. This led to persistent congestion"; "a latent issue prevented these clients from adequately backing off during this event." |
| 20 | AWS | same | postmortem | 2021-12 | 2026-08-29 | https://aws.amazon.com/message/12721/ | The feedback loop blinded the operators who had to break it | "This congestion immediately impacted the availability of real-time monitoring data for our internal operations teams, which impaired their ability to find the source of congestion and resolve it." |
| 21 | AWS | Summary of the AWS Service Event in the Northern Virginia Region, Oct 19-20 2025 | postmortem | 2025-10 | 2026-08-29 | https://aws.amazon.com/message/101925/ | AWS names the state directly, and it began after the trigger was fixed | "At this point, DWFM had entered a state of congestive collapse and was unable to make forward progress." DynamoDB DNS was restored at 2:25 AM PDT; the collapse began then. |
| 22 | AWS | same | postmortem | 2025-10 | 2026-08-29 | https://aws.amazon.com/message/101925/ | Recovery was manual throttling plus restarts, and the durable fix is queue-aware admission | at 4:14 AM engineers "throttled incoming work and began selective restarts of DWFM hosts"; corrective action: "improving throttling mechanism to rate-limit work based on queue size" |
| 23 | Slack | Slack's Outage on January 4th 2021 | postmortem | 2021-02 | 2026-08-29 | https://slack.engineering/slacks-outage-on-january-4th-2021/ | The scaling response became a second failure, and panic mode was what stabilised serving | 1,200 servers requested between 07:01 and 07:15 PST; provision-service hit "the Linux open files limit" and "exceeded an AWS quota limit"; "our load balancers have a feature called 'panic mode' which balances requests across all instances when many are failing health checks" |
| 24 | GitHub | The August 17 outage, and the work ahead (Vlad Fedorov, CTO) | postmortem | 2026-08-20 | 2026-08-29 | https://github.blog/news-insights/company-news/the-august-17-outage-and-the-work-ahead/ | The 2026 remediation is the 2016 prescription | "First, we are applying consistent retry limits, retry budgets, and variable timeouts across service-to-service interactions to prevent retry storms and cascading load." |
| 25 | GitHub | 2026-08-17 incident thread, official incident summary | postmortem | 2026-08-18 | 2026-08-29 | https://github.com/orgs/community/discussions/205164 | A client-side retry bug produced roughly 10x amplification and delayed recovery | "Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS"; "a latent retry bug in VS Code that amplified traffic by approximately 10x"; "Pausing HAProxy on those nodes simultaneously produced immediate broad recovery." |
| 26 | AWS (Marc Brooker) | Timeouts, retries, and backoff with jitter (Amazon Builders' Library) | vendor | undated, retrieved 2026-08-29 | 2026-08-29 | https://d1.awsstatic.com/onedam/marketing-channels/website/aws/en_US/product-categories/developer-tools/approved/pdfs/timeouts-retries-and-backoff-with-jitter.pdf | Layered retries multiply; five layers at three attempts is 243x | "It ends with a query to a database, and three retries at each layer… If each layer retries independently, the load on the database will increase 243x, making it unlikely to ever recover." |
| 27 | AWS (Marc Brooker) | same | vendor | undated | 2026-08-29 | https://d1.awsstatic.com/onedam/marketing-channels/website/aws/en_US/product-categories/developer-tools/approved/pdfs/timeouts-retries-and-backoff-with-jitter.pdf | AWS rejects circuit breakers in favour of token buckets, and says why | "circuit breakers introduce modal behavior into systems that can be difficult to test, and can introduce significant addition time to recovery. We have found that we can mitigate this risk by limiting retries locally using a token bucket… AWS added this behavior to the AWS SDK in 2016." |
| 28 | AWS (Marc Brooker) | same | vendor | undated | 2026-08-29 | https://d1.awsstatic.com/onedam/marketing-channels/website/aws/en_US/product-categories/developer-tools/approved/pdfs/timeouts-retries-and-backoff-with-jitter.pdf | Retries are a client asserting priority over other clients | "Retries are 'selfish.'… Retries are a way for clients to assert the importance of their request and demand that the service spend more of its resources to handle it." |
| 29 | AWS (Marc Brooker) | Exponential Backoff And Jitter (AWS Architecture Blog) | blog | 2015-03-04 | 2026-08-29 | https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ | Jitter, not backoff, is what decorrelates a retrying herd | full jitter sleep = random(0, min(cap, base * 2 ** attempt)); decorrelated jitter sleep = min(cap, random(0, last_sleep * 3)); "with 100 contending clients, we've reduced our call count by more than half" |
| 30 | Site Reliability Engineering, ch. 22: Addressing Cascading Failures | blog | 2016 | 2026-08-29 | https://sre.google/sre-book/addressing-cascading-failures/ | Layered retries multiply, and the fix is a per-process retry budget | "a single request at the highest layer may produce a number of attempts as large as the product of the number of attempts at each layer"; "only allow 60 retries per minute in a process, and if the retry budget is exceeded, don't retry; just fail the request" | |
| 31 | same | blog | 2016 | 2026-08-29 | https://sre.google/sre-book/addressing-cascading-failures/ | Recovery is drop load, let servers become healthy, then ramp | "Reducing load enough so that the crashing stops… Allowing the majority of the servers to become healthy. Gradually ramping up the load." | |
| 32 | Envoy | Circuit breaking, architecture overview | vendor | current, retrieved 2026-08-29 | 2026-08-29 | https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/circuit_breaking | Envoy recommends budgets over fixed retry counts | "In general we recommend using retry budgets; however, if static circuit breaking is preferred it should aggressively circuit break retries." |
| 33 | Envoy | circuit_breaker.proto API reference | source | current | 2026-08-29 | https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/circuit_breaker.proto | The shipped defaults: 20% budget, floor of 3, or 3 fixed retries | budget_percent default 20%, "limit on concurrent retries as a percentage of the sum of active requests and active pending requests"; min_retry_concurrency default 3; max_retries default 3 |
| 34 | Envoy | Issue #30205, circuit breaker retry budgets count retries inconsistently | source | 2023-10-13, closed as not planned | 2026-08-29 | https://github.com/envoyproxy/envoy/issues/30205 | The recommended mechanism has an accounting gap: retries in backoff count against the limit but not in the denominator | budgets count "number of outstanding retries, either active, pending, in backoff, or waiting for rate limiting" while the limit uses only "(num_pending_requests + num_active_requests)"; even a 100% budget can reject |
| 35 | Envoy | PR #30738, retry budgets count scheduled retries towards budget | source | 2023-11-06, closed unmerged 2024-01-02 | 2026-08-29 | https://github.com/envoyproxy/envoy/pull/30738 | First attempt to fix the accounting was closed in favour of a redesign | reviewer adisuissa: "The addition of scheduled retries may break current users."; closed with "Closing in favor of #31565" |
| 36 | Envoy | PR #43792, attempt admission control extension point for circuit breaker | source | 2026-03-05, closed unmerged 2026-05-21 | 2026-08-29 | https://github.com/envoyproxy/envoy/pull/43792 | Prioritised shedding of retries exists internally at contributors' employers but not upstream; the PR died on stalebot | KBaichoo: used internally for "prioritized shedding of retries to avoid retry storms, as well as informing envoy of application retries"; markdroth argued for "a universal extension point for circuit breaking" |
| 37 | gRPC | gRFC A6: client retries | adr | 2017 onward, current | 2026-08-29 | https://github.com/grpc/proposal/blob/master/A6-client-retries.md | gRPC's throttle is a per-server token bucket that disables retries and hedging wholesale | "gRPC prevents server overload due to retries and hedged RPCs by disabling these policies when the client's ratio of failures to successes passes a certain threshold"; failures decrement by 1, successes add tokenRatio, retries blocked below maxTokens / 2 |
| 38 | gRPC | same | adr | current | 2026-08-29 | https://github.com/grpc/proposal/blob/master/A6-client-retries.md | The server can tell the client not to retry at all | grpc-retry-pushback-ms; "If the value for pushback is negative or unparseble, then it will be seen as the server asking the client not to retry at all." |
| 39 | gRPC | grpc-java issue #11274, retry throttling not respected with default service config | source | 2024-06-10 | 2026-08-29 | https://github.com/grpc/grpc-java/issues/11274 | The throttle can be silently inactive depending on how the config arrives | "The throttling policy is only being used if it comes from the NameResolver." With a default service config and no resolver-supplied config, throttling is bypassed and all retries execute. |
| 40 | Kubernetes | KEP-1040, API Priority and Fairness | adr | graduated, current | 2026-08-29 | https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/1040-priority-and-fairness/README.md | Replacing a global inflight cap with priority levels, fair queuing and shuffle sharding | "there can be undesirable scenarios where one subset of the request load crowds out other parts of the request load"; shuffle sharding gives collision probability of roughly "one in 5.4 billion"; "This request is rejected if and only if the total number of requests waiting in its queue is at least the configured limit." |
| 41 | Netflix | concurrency-limits, README | source | current | 2026-08-29 | https://github.com/Netflix/concurrency-limits/blob/main/README.md | Static limits go stale; treat the concurrency limit as a congestion window | "in large distributed systems that auto-scale this value quickly goes out of date and the service falls over by becoming non-responsive"; "we borrow from common TCP congestion control algorithms by equating a system's concurrency limit to a TCP congestion window" |
| 42 | Netflix | same | source | current | 2026-08-29 | https://github.com/Netflix/concurrency-limits/blob/main/README.md | The server limiter exists specifically to survive retry storms | "protect the server from either increased client traffic (batch apps or retry storms) or latency spikes from a dependent service" by "rejecting excess traffic" |
| 43 | Uber | Cinnamon: using century-old tech to build a mean load shedder | blog | 2023-11-22 | 2026-08-29 | https://www.uber.com/en-IN/blog/cinnamon-using-century-old-tech-to-build-a-mean-load-shedder/ | Manually tuned concurrency limits do not survive thousands of services | QALM required per-service caller priorities and max concurrent request limits, "expensive to maintain"; "setting the right maximum number of concurrent requests is difficult, as it can really change fairly rapidly" |
| 44 | Uber | same | blog | 2023-11-22 | 2026-08-29 | https://www.uber.com/en-IN/blog/cinnamon-using-century-old-tech-to-build-a-mean-load-shedder/ | Goodput collapse under a badly tuned shedder, measured | at 3,000 RPS QALM total goodput fell to 552 requests, about 40% of capacity, while Cinnamon stayed near capacity; Cinnamon overhead about 1 microsecond per request; tier-1 latency up about 50% at 300% overload |
| 45 | Meta (Ben Maurer) | Fail at Scale, ACM Queue, as summarised by the morning paper | blog | 2015-11-19 | 2026-08-29 | https://blog.acolyer.org/2015/11/19/fail-at-scale-controlling-queue-delay/ | Meta's queue-delay controls: CoDel timeouts and adaptive LIFO | CoDel target and interval "5ms and 100ms respectively seems to work well"; "requests are normally processed FIFO, but once a queue starts to form services switch to LIFO ordering"; per-service outstanding-request caps on the client |
| 46 | Meta (Nathan Bronson) | Solving the mystery of link imbalance: a metastable failure state at scale | blog | 2014-11-14 | 2026-08-29 | https://engineering.fb.com/2014/11/14/production-engineering/solving-the-mystery-of-link-imbalance-a-metastable-failure-state-at-scale/ | A feedback loop can hide inside a connection-pool policy and defeat two years of investigation | MRU pool with a 10-second idle timeout meant slow queries "lose the race reliably" and became most-recently-used; two years to root-cause; fixed by switching to LRU with a maximum connection age |
| 47 | Marc Brooker | Metastability and Distributed Systems | blog | 2021-05-24 | 2026-08-29 | https://brooker.co.za/blog/2021/05/24/metastable.html | Optimising the common case is what builds the loop | adding retries "can make systems more vulnerable, by converting small outages into sudden (and metastable) periods of internal retry storms"; systems sit "Up, but down. Working, but broken" |
| 48 | Grab (Michael Cartmell) | Designing Resilient Systems Beyond Retries, Part 1: Rate-Limiting | blog | 2019-03-20 | 2026-08-29 | https://engineering.grab.com/beyond-retries-part-1 | A server cannot outsource its own protection to its clients | "it is never a good idea for the server to depend on its clients for resiliency. The circuit-breaker could fail or simply be bypassed, and the server would have to deal with all requests the client makes." |
| 49 | Cloudflare | Code Orange: Fail Small, our resilience plan following recent incidents | blog | 2026 | 2026-08-29 | https://blog.cloudflare.com/fail-small-resilience-plan/ | The structural response to a global-blast-radius incident is staged propagation and validated defaults | "there should have been a sane set of validated defaults which would have allowed traffic to pass through our network"; Health Mediated Deployment for configuration changes |
| 50 | USENIX / Penn State et al. | Metastable Failures in the Wild, OSDI '22 conference presentation | talk | 2022-07-11 | 2026-08-29 | https://www.usenix.org/conference/osdi22/presentation/huang-lexiang | The conference record of the study, with slides and video | abstract: "we study the prevalence of such failures in the wild by scouring over publicly available incident reports from many organizations, ranging from hyperscalers to small companies" |
| 51 | FluxNinja | Mastering Chaos: Observability-Driven Prioritized Load Shedding, SREcon23 APAC | talk | 2023-06-16 | 2026-08-29 | https://www.usenix.org/conference/srecon23apac/presentation/gill | Practitioner framing of the same loop as a Little's law problem, with the permanent-overload picture | slide 4: "Every service has an inherent concurrency limit. For a service to remain stable, concurrent requests must be limited"; slide 9 "Retry storm: permanent overload… System is in a state of permanent overload" after "Capacity restored to the original level" |
| 52 | FluxNinja | same, slide deck PDF | talk | 2023-06-16 | 2026-08-29 | https://www.usenix.org/system/files/srecon23apac_slides-gill_0.pdf | Slides as cited above | slide 8: "Retries put even greater pressure on the database" |
Tier mix
postmortem 12, source 9, adr 4, paper 8, blog 11, talk 3, vendor 5. 52 rows over 30 distinct artefacts, 16 distinct hosts.
Where the record runs out
- No organisation has published a measured before-and-after for introducing a retry budget: what the amplification factor was, what it became, what it cost in availability during ordinary transient faults. Envoy ships a 20% default with no published derivation.
- No public account describes deadline propagation and retry budgets interacting, though both are recommended by the same sources.
- Nobody has published the cost of the headroom. Running in the stable rather than the vulnerable state is a capacity purchase, and no public source prices it.
- The OSDI '22 sample is 22 incidents that were public and detailed enough to classify. It is a sample of what large companies chose to write down, not of what happens.