The retry is the outage: retry storms and metastable failure

The territory

The problem, stated without naming the technology: a system that was serving correctly keeps not serving after the thing that broke it has been repaired, and no amount of waiting fixes it.

243×

Load increase on a database when five service layers each retry three times, independently

Amazon Builders' Library

>50%

Of 22 studied metastable failures were sustained by retry policy, more than any other mechanism

Huang et al., OSDI '22

1.5–73.5h

Observed outage range once a system enters the self-sustaining state

Huang et al., OSDI '22

10×

Traffic amplification from one client retry bug during GitHub's recovery, 17 August 2026

GitHub incident summary

On 20 September 2015 a network blip in one AWS availability zone lasted a few minutes. DynamoDB in us-east-1 was impaired for four and a half hours. The blip was not the outage. What kept the service down was that its storage servers, unable to fetch their partition membership data, retried, and the retries held the metadata service at a load it could not clear. AWS's own write-up is blunt about it: "Unavailable servers continued to retry requests for membership data, maintaining high load on the metadata service." The team could not add capacity to fix it, because adding capacity required successfully calling the service that was saturated. At 5:06 AM PDT they gave up on repair and paused the traffic instead, and that is what ended the outage.

Eleven years later, on 17 August 2026, GitHub was down for seven hours and forty-seven minutes. The trigger was different: an Istio sidecar hit its concurrency limits and four HAProxy nodes exhausted their flow limits. The shape was identical. GitHub's incident summary records that Copilot Token Service traffic "increased from a normal 7–9K RPS to 70–100K RPS" because "a latent retry bug in VS Code" amplified traffic roughly tenfold, and that the amplification "caused delayed recovery". Recovery again came from removing load rather than adding capacity: "Pausing HAProxy on those nodes simultaneously produced immediate broad recovery."

The finding worth your Thursday

The single most useful thing in this material is not the mechanism, which is well known. It is that the remediation from one incident has repeatedly produced the next one. Huang and colleagues document the chain: after AWS SimpleDB failed in June 2014, engineers concluded that storage servers giving up on the locking service was why recovery went badly, and decided "that servers must continue to retry the locking service instead of giving up." Fifteen months later the DynamoDB outage above happened because storage nodes, in the paper's words, "did not back out of retrying to get updated membership data." Spotify did the same thing in a different register: after one incident driven by retries they added detailed logging to the error path so they could understand it, and in the next incident "the additional logging after a load spike and initial retries increased the cost of each retry." The post-incident action item is where this failure class is manufactured.

Bronson, Aghayev, Charapko and Zhu named the pattern in 2021 and gave it a state machine that has since become the common vocabulary. A system is stable when it can absorb a temporary overload and return by itself. It is vulnerable when it is serving fine but a trigger would push it somewhere it cannot return from. It is metastable when goodput has collapsed and a sustaining feedback loop keeps it collapsed after the trigger is gone. The uncomfortable part of their paper is not the taxonomy, it is this sentence: "many production systems choose to run in the vulnerable state all the time because it has much higher efficiency than the stable state." The boundary between stable and vulnerable is what they call hidden capacity, and almost nobody measures it, because it is invisible while everything is working.

Figure 1 · Three states, and only one transition an operator controls

Notice that there is no arrow out of the metastable state that the system can take by itself. Every published recovery in this guide is the dashed one. State model from Bronson et al., HotOS '21; the recovery arrow is corroborated by Huang et al., OSDI '22, which found load shedding in over half of the mitigations.

Diagram source

Who has published on this, and what they run

The public record is unusually good for a topic this awkward, because the largest operators publish incident reports and two academic groups went through them systematically. AWS has four incidents in the OSDI '22 sample and a fifth, in October 2025, in which the phrase "congestive collapse" appears in AWS's own prose. Google contributes four incidents and the SRE book chapter that first prescribed retry budgets in public. Meta contributes two first-hand engineering accounts, one of which took two years to root-cause. Azure contributes four incidents, IBM one that ran for just over three days, Spotify the pair described above, and Elastic, Wikimedia, CircleCI, Cassandra and Ably one each. Uber, Netflix, Grab and DoorDash have published the countermeasures rather than the incidents. The client libraries and proxies that most enterprises actually deploy, gRPC, Envoy and the AWS SDKs, all ship a version of the same control, and their issue trackers contain the argument about whether it works.

What this guide does not cover

Not covered: idempotency and exactly-once semantics, which decide whether a retry is safe rather than whether it is affordable; distributed tracing and detection tooling; chaos engineering practice; and the organisational half of incident response, which is the subject of the sibling guide in this collection. Also out of scope: denial-of-service and abuse traffic. That is an adversarial load problem and resolves when the attacker stops, which is precisely what makes it not metastable.

The overload control stack, as five layers that most teams have two of

No single organisation publishes this as one diagram. It is the union of what Google, Meta, Uber, Netflix, AWS, Kubernetes, gRPC and Envoy each described separately, and the striking thing is how consistent the layers are and how inconsistently they are deployed.

Start with the arithmetic, because it decides everything downstream. Retries multiply rather than add. Google's SRE book states the general form: "a single request at the highest layer may produce a number of attempts as large as the product of the number of attempts at each layer." Marc Brooker's Builders' Library article puts a number on the configuration most enterprise stacks actually have, five service hops with three attempts configured at each, and it is 243 times the intended load. That figure deserves to be read twice, because three retries is a default, not an aggressive setting, and five hops is a modest microservice call graph. Nobody chose 243. It is what a stack does when eight different teams each independently make a locally sensible choice.

Figure 2 · Why nobody configured 243

Each layer's three attempts multiply against the layer below. Arithmetic from Brooker, Amazon Builders' Library; the general statement is in Google's SRE book, chapter 22. Backoff changes when these arrive, not how many there are.

Diagram source

The second thing to internalise is that backoff does not cap this. Exponential backoff with jitter, which is the intervention almost every team reaches for first, changes the arrival schedule of the 243 queries. It does not change the number 243. Brooker's own 2015 post that popularised jitter is explicit about what it buys, which is decorrelation of a herd: "with 100 contending clients, we've reduced our call count by more than half" for a contention workload. That is a real and worthwhile win, and it is a different win from bounding total work. Capped exponential backoff also has a failure mode of its own that the Builders' Library names: "Now all of the clients are retrying constantly at the capped rate." Backoff is a scheduling policy. Only a budget is a bound.

Figure 3 · Reference architecture: the five places a limit can live

The common shape across the published systems. Layers 1 and 5 are nearly universal; layers 2, 3 and 4 are where organisations diverge, and each divergence is a decision in the next section. Attribution for each box is in the cards below.

Diagram source

Layer 1: bound the client, not just the schedule

gRPC's retry design is a per-server token bucket. Failures decrement the count by one, successes add a fractional tokenRatio, and when the count falls below maxTokens / 2 the client stops retrying and stops hedging entirely. The stated purpose is exactly this failure class: gRPC "prevents server overload due to retries and hedged RPCs by disabling these policies when the client's ratio of failures to successes passes a certain threshold." AWS added the equivalent to its SDKs in 2016.

Runs this way at: gRPC (gRFC A6), AWS SDKs

Layer 2: a ratio, not a count

Envoy's retry budget expresses the limit as a percentage of live work rather than a fixed number, defaulting to 20% of active plus pending requests with a floor of three. The documentation is unambiguous about preference: "In general we recommend using retry budgets; however, if static circuit breaking is preferred it should aggressively circuit break retries." A ratio scales with traffic; a fixed count of three is either useless at peak or crippling at trough.

Runs this way at: Envoy, and in a per-process form in Google's SRE practice

Layer 3: treat concurrency as a congestion window

Netflix's library states the problem with the obvious approach plainly: a hand-set limit "quickly goes out of date and the service falls over by becoming non-responsive", so they "borrow from common TCP congestion control algorithms by equating a system's concurrency limit to a TCP congestion window." Meta's queue-side equivalent is CoDel with a 5 ms target and 100 ms interval, plus adaptive LIFO: FIFO normally, LIFO once a queue forms, so that at least some requests beat their deadline.

Runs this way at: Netflix, Uber, Meta

Layer 4: decide in advance who loses

Kubernetes replaced a single global in-flight cap with priority levels, fair queuing and shuffle sharding, because with one cap "there can be undesirable scenarios where one subset of the request load crowds out other parts of the request load". Node heartbeats losing to a runaway controller is the scenario. Uber propagates caller priority through tracing headers so a tier-1 request outranks a batch job several hops down. Both are deliberate answers to a question uniform shedding answers by accident.

Runs this way at: Kubernetes, Uber

Layer 5: the lever that actually works

Every recovery in the failure catalogue below is this layer. AWS paused metadata requests in 2015 and throttled DWFM work in 2025. GitHub paused HAProxy on four nodes in 2026 and got "immediate broad recovery". Google's SRE book gives the sequence: reduce load until the crashing stops, let the majority of servers become healthy, then ramp gradually. The interesting question is not whether you will need this lever. It is whether it exists and whether anyone has ever pulled it.

Used at: AWS 2015, AWS 2025, GitHub 2026

What is missing from almost every real stack

Reconstruction rather than report: across the twelve incidents, the layer that was present was layer 1 and the layer that was pulled was layer 5. Layers 2, 3 and 4 appear in the countermeasure literature and in the post-incident action items, rarely in the incident narrative itself. GitHub's own remediation list, published in August 2026, is a commitment to build layer 2: "applying consistent retry limits, retry budgets, and variable timeouts across service-to-service interactions".

Inferred from: the twelve incidents in the ledger, plus GitHub's remediation

The sustaining effect is not always a retry

Fixating on retries is a trap, and the primary sources are careful about it. Bronson and colleagues catalogue four mechanisms from Meta production, and only one is a retry. A look-aside cache is the second: losing a cache with a 90% hit rate is a tenfold query amplification against the database, and the look-aside shape makes recovery structurally impossible, because the application that would refill the cache is the same one timing out. Their observation on that point is the most actionable sentence in the paper for anyone choosing a caching pattern: prioritising cache fill over serving "is unenforceable with a look-aside cache but trivial with a read-through cache". Third is slow error handling, where the failure path captures a stack trace, does a reverse DNS lookup and writes to disk, so each failure costs more than each success. Fourth is their link imbalance case, where a most-recently-used connection pool systematically preferred the slowest network link, because slow queries finish last and therefore land on top of the stack. That one defeated Meta for two years and multiple vendors of switch firmware, and the fix was one line changing the pool to least-recently-used.

The common structure across all four is a mechanism that increases work per unit of demand when the system is unhealthy. That phrasing is the test to apply to your own design, and it catches things a retry-focused review misses: health checks that fail and trigger instance replacement, autoscaling that adds cold nodes which are net consumers before they are producers, leader elections that churn under load, and diagnostic logging on the error path.

The decisions that matter

Six forks where the published record contains an actual argument, including one where two of the most credible sources disagree with each other.

Figure 4 · Where to put the limit

Every terminal node is an action, not a preference. The left-hand branch is the one most teams skip, and it is the cheapest. Derived from the decision blocks below.

Diagram source

Decision 1: how many layers of the call graph are allowed to retry?

Chosen

Exactly one, at a single point in the stack. AWS's stated best practice "for low-cost control-plane and data-plane operations".
Google states the same rule as "avoid amplifying retries by issuing retries at multiple levels".

Rejected

Retry wherever a client library makes it easy, which is the de facto default.
Rejected because attempts multiply: five layers at three attempts is 243 times the load.

Flips when

The call is expensive to redo from the top. AWS notes "retrying at the highest layer of the stack may waste work from previous calls". Then retry lower down, and put a budget on that layer rather than a count.

Decision 2: circuit breaker, or token bucket?

Chosen

Token bucket, at AWS. It degrades continuously: all calls retry while tokens remain, then retries continue at a fixed low rate.
Shipped in the AWS SDKs since 2016, so most AWS customers have it without knowing.

Rejected

The circuit breaker, explicitly. AWS's stated reason: breakers "introduce modal behavior into systems that can be difficult to test, and can introduce significant addition time to recovery".

Flips when

The dependency fails in a binary way rather than by slowing down, and you can exercise the open state in a test environment. A breaker is a good answer to "it is gone" and a poor answer to "it is slow".

Decision 3: whose job is it to protect the server?

Chosen

The server's, at Grab and Netflix. Grab's reasoning: "it is never a good idea for the server to depend on its clients for resiliency. The circuit-breaker could fail or simply be bypassed."
Netflix's server limiter exists to survive "batch apps or retry storms" it does not control.

Rejected

Client-side discipline alone, which is what gRPC and the AWS SDK provide.
GitHub 2026 is the empirical argument: the amplifying client was VS Code, running on other people's laptops.

Flips when

You genuinely own every caller and the call graph is internal. Then client budgets are cheaper and preserve more goodput than server-side rejection, because the work is never sent.

Decision 4: shed load, or add capacity?

Chosen

Shed load. It was the mitigation in over half of the 22 studied incidents, and it is the only lever with a published record of working.

Rejected

Scale out during the event. Slack tried to add 1,200 servers in fourteen minutes on 4 January 2021 and the provisioning service itself hit the Linux open files limit and an AWS quota.
AWS's Kinesis fleet could only be restarted "at the rate of a few hundred per hour".

Flips when

Never, during the event. Capacity is a lever for the vulnerable state, not the metastable one. Bronson and colleagues: "unless a stateful system is specifically designed to provide zero-impact elasticity, reconfiguration will reduce capacity in the short term."

Decision 5: uniform shedding, or priority shedding?

Chosen

Priority, at Kubernetes and Uber. Kubernetes uses priority levels plus fair queuing plus shuffle sharding, with a flow collision probability of roughly one in 5.4 billion.
Uber propagates caller priority through tracing headers so it survives several hops.

Rejected

A single global in-flight cap, which Kubernetes had and abandoned: "one subset of the request load crowds out other parts of the request load".

Flips when

You cannot propagate priority end to end. A priority scheme that covers some resources and not others is worse than none: Bronson reports a geo-distributed system with "a worst-case work amplification of over 100x" that had a sophisticated priority system and still fell over.

Decision 6: a fixed concurrency limit, or an adaptive one?

Chosen

Adaptive, at Netflix and Uber, both borrowing from TCP congestion control. Uber replaced its hand-tuned shedder with a PID controller plus a TCP-Vegas variant and got zero configuration per service.

Rejected

Per-service tuned limits. Uber found them "expensive to maintain" across thousands of services, and measured the cost of getting them wrong: at 3,000 RPS their old shedder delivered 552 successful requests, about 40% of capacity.

Flips when

You have tens rather than thousands of services on homogeneous hardware. Then a static cap you alarm on is simpler, testable, and does not have a control loop of its own to debug at 3 AM.

Where the sources disagree

AWS and the wider practitioner literature are in direct conflict about circuit breakers. Nygard's pattern is in the HotOS paper's own list of countermeasures, it is the default in most resilience libraries, and it is what most enterprise architecture review boards expect to see on the diagram. AWS says in print that it rejected it, and gives a specific engineering reason rather than a preference: modal behaviour is hard to test and lengthens recovery. Both positions are defensible and they are answering different questions. A breaker optimises for not sending doomed work to a dependency that is down. A token bucket optimises for degrading smoothly against a dependency that is slow. Metastable failure is a slowness problem, which is why the operator of the largest slowness problem in the industry landed where it did. If your architecture board wants a breaker on the diagram, the productive move is to ask what its half-open probe rate is, because that is the token bucket reappearing under a different name.

Decision	Chosen	Rejected	Because	Evidence
Layers allowed to retry	One	Every layer	Attempts multiply to 243x at five hops	AWS, Google SRE
Retry limiter shape	Token bucket or ratio budget	Circuit breaker	Modal behaviour is hard to test and slows recovery	AWS, Envoy
Budget expressed as	Percentage of active work, default 20%	Fixed count, default 3	A ratio scales with traffic; a count does not	Envoy proto
Protection sits with	The server	Client discipline alone	Clients can be bypassed, buggy, or on someone's laptop	Grab, GitHub 2026
Recovery lever	Reduce load, then ramp	Add capacity mid-incident	Reconfiguration consumes capacity before it adds any	Slack, AWS Kinesis
Shedding policy	Priority plus fair queuing	One global in-flight cap	A single cap lets one flow crowd out heartbeats	KEP-1040
Concurrency limit	Adaptive, from queue delay	Hand-tuned per service	Tuned limits go stale; the cost of stale is measured at 40% of capacity	Netflix, Uber

The countermeasure has its own bug report

Retry budgets are the recommended mechanism, and the most-deployed implementation of them has an accounting problem that has been open since October 2023 and is not fixed. Envoy issue #30205 documents it: the budget counts retries that are "active, pending, in backoff, or waiting for rate limiting", but computes the limit from (num_pending_requests + num_active_requests) only. Retries sitting in backoff are therefore in the numerator and not the denominator, with the consequence that even a budget set to 100% can reject retries. The same issue reports that HTTP/1 and HTTP/3 upstreams count completed requests as active while HTTP/2 does not, so the effective budget depends on the protocol.

What happened next is the part worth reading. The issue was closed as not planned. Three pull requests tried to fix or generalise the mechanism and none of them merged: #30738 in November 2023, closed in January 2024 in favour of a redesign; the redesign PRs closed in February 2024; and #43792 in March 2026, which stalebot closed in May 2026 after the maintainers could not agree whether retry admission control should be its own extension point or a case of a general one. In that last thread a Google engineer argued for "a universal extension point for circuit breaking" rather than another field on the thresholds message, and an Envoy contributor mentioned that their employer already runs an internal implementation for "prioritized shedding of retries to avoid retry storms, as well as informing envoy of application retries". Read that plainly: the large operators have built the good version privately, the public version has a known accounting gap, and the design argument about how to close it has been running for close to three years.

gRPC has a matching lesson at the other end. grpc-java issue #11274, filed in June 2024, reports that retry throttling is silently inactive when the retry policy arrives via a default service config rather than from the name resolver, because "the throttling policy is only being used if it comes from the NameResolver". A safety mechanism that is configured and does nothing is worse than one that is absent, because it has been checked off in the design review. If you rely on a budget, the acceptance test is not that it is configured. It is that you can produce the rejection metric on demand in a load test.

What broke in production

Twelve published incidents, grouped by sustaining mechanism rather than by company, because the mechanism is what transfers and the company is what does not.

Figure 5 · The failure path, with the two moments that decide the outage length

DynamoDB, 20 September 2015. The network blip is repaired long before minute thirty; the outage runs to minute two hundred and seventy. Notice that the operator's capacity fix requires the saturated service to answer. Reconstructed from AWS's summary.

Diagram source

Class A: retry amplification against a shared dependency

Postmortem

AWS DynamoDB metadata service, September 2015

AssumptionThat a brief network disruption is a transient the fleet absorbs, and that membership requests are cheap.

What happenedGlobal secondary indexes had quietly grown membership payloads until processing time approached the retrieval allowance. Simultaneous requests after the blip pushed processing past the limit, and unavailable servers retried, holding the load up.

Blast radiusAbout 4h 51m from trigger to recovery; roughly 55% error rates at the peak; SQS, EC2 Auto Scaling, CloudWatch and the AWS Console affected.

FixCapacity, stricter monitoring of membership size, a lower request rate from storage nodes and a longer allowance, and structurally: many instances of the metadata service each serving a portion of the fleet.

Design ruleIf your control plane is a single shared service, its blast radius is your whole fleet, and its own administrative API must be on a path that overload cannot block.

SourceAWS summary, 20 September 2015

Postmortem

GitHub, 17 August 2026

AssumptionThat a client retry loop shipped in an editor is bounded, and that recovery capacity only has to serve normal traffic.

What happenedAn Istio sidecar hit its concurrency limits and four HAProxy nodes exhausted their flow limits, degrading the auth path. A latent retry bug in VS Code amplified traffic tenfold, taking Copilot Token Service from 7 to 9K RPS to 70 to 100K RPS and preventing recovery.

Blast radius7h 47m; about 20% error rates on web and API, about 50% on archive and raw content downloads; Issues, Pull Requests, Actions and Copilot.

FixPausing HAProxy on the affected nodes gave "immediate broad recovery". Committed remediation: consistent retry limits, retry budgets and variable timeouts across service-to-service calls.

Design ruleCapacity planning for recovery is a separate exercise from capacity planning for peak, and the input is peak times your worst client's amplification factor, which you do not control.

SourceIncident summary, CTO write-up

Postmortem

AWS us-east-1 internal network, December 2021

AssumptionThat the well-tested backoff behaviour in the internal networking clients would engage.

What happenedAn automated scaling activity produced a connection surge that overwhelmed devices between two internal networks. Latency and errors produced "even more connection attempts and retries", and "a latent issue prevented these clients from adequately backing off during this event".

Blast radiusAbout 9.3 hours across DynamoDB, EC2, Fargate, RDS, EMR, Workspaces, the Console and internal DNS.

FixTraffic isolation onto dedicated devices, disabling the triggering automation, and shipping the backoff fix.

Design ruleBackoff logic that has never been exercised under real congestion is a claim, not a control. Test it by inducing congestion, not by unit-testing the sleep calculation.

SourceAWS summary, 7 December 2021

Paper

IBM Cloud DNS and crypto services, June 2021

AssumptionNot published in enough detail to name it, which is itself the finding.

What happenedA software bug triggered a failure sustained by retries, per the classification in the OSDI '22 survey. IBM does not provide a direct public incident link, so the mechanism is known only at this resolution.

Blast radius73.53 hours, the longest in the sample, across private DNS, hyper protect crypto services, Cloudant DNS and Cloud Shell.

FixLoad shedding, a policy change and a hotfix for the trigger.

Design ruleDuration in this failure class is set by how long it takes a human to identify the loop, not by the difficulty of the trigger. Three days is what happens when nobody on the call has the vocabulary.

SourceHuang et al., OSDI '22, Table 1

Class B: recovery work that exceeds serving capacity

Postmortem

AWS EC2 DWFM, October 2025

AssumptionThat once the underlying DNS defect was repaired, dependent subsystems would reconverge on their own.

What happenedDynamoDB DNS was restored at 02:25 PDT. DWFM then tried to re-establish leases with thousands of droplets at once, work timed out, more work queued, and in AWS's own words "DWFM had entered a state of congestive collapse and was unable to make forward progress".

Blast radiusEC2 launches impaired until 13:50 PDT, roughly eleven hours after the trigger was fixed; knock-on backlogs in Lambda, NLB and Redshift lasting into 21 October.

FixThrottling incoming work plus selective host restarts, then relaxing throttles. Committed change: "improving throttling mechanism to rate-limit work based on queue size".

Design ruleReconvergence after an outage is a workload, and it is usually larger than peak. Size admission control for the recovery burst, and make queue depth, not CPU, the input.

SourceAWS summary, 19 to 20 October 2025

Postmortem

AWS Kinesis front-end fleet, November 2020

AssumptionThat a small capacity addition is a routine, reversible operation.

What happenedThe addition pushed every server past an operating system thread limit. Cache construction failed, leaving front-end servers with "useless shard-maps". Restarting the fleet was itself the bottleneck: each server rebuilding its shard-map contends with serving traffic.

Blast radiusAbout 17 hours from first alarm to full recovery; servers could be returned "at the rate of a few hundred per hour", and fleet membership propagation "takes up to an hour".

FixLarger servers to cut thread counts, thread monitoring, raised limits, and cellularisation of the front-end fleet.

Design ruleMeasure how long a cold start of the whole fleet takes, in production, once. That number is the floor on every outage you will ever have in that system.

SourceAWS summary, 25 November 2020

Postmortem

Slack, 4 January 2021

AssumptionThat autoscaling is the response to saturation, and that the provisioning path is independent of the saturated path.

What happenedAn overloaded transit gateway degraded the network on the first Monday after the holidays, when client caches were cold. Two scaling signals fired at once and 1,200 servers were requested between 07:01 and 07:15. The provisioning service, talking over the same degraded network, hit the Linux open files limit and an AWS quota, and broken instances filled the autoscaling groups.

Blast radiusMessage success rate fell to about 99% against a normal better than 99.999%.

FixLoad balancer panic mode spread requests across all instances regardless of health checks, which stabilised serving. Afterwards: dashboards in the same VPC as their database, regular load testing of the provisioning service, and a review of health checking and autoscaling.

Design ruleHealth checks plus autoscaling form a feedback loop. When most instances are failing checks because the dependency is slow, removing them is the wrong action, which is exactly what panic mode encodes.

SourceSlack Engineering

Paper

The look-aside cache that cannot refill itself

AssumptionThat a 90% hit rate means the database is sized for 10% of traffic.

What happenedLosing the cache contents is a tenfold query amplification. The application then times out on database queries, and because it is the application that populates a look-aside cache, a timed-out query never fills the cache. Hit rate stays at zero.

Blast radiusPresented as a generalised case from Meta production rather than a single dated incident. Advertised capacity 3,000 QPS, hidden capacity 300 QPS.

FixA read-through cache, where a permissive timeout on the cache's own database query fills the cache even after the application has given up.

Design ruleChoose read-through over look-aside wherever the cache fronts a slower dependency. The reason is not performance, it is that only read-through lets you prioritise refill over serving during recovery.

SourceBronson et al., HotOS '21, section 2.2

Class C: the loop the post-incident review installed

Paper

SimpleDB 2014 to DynamoDB 2015: the remediation that armed the next outage

AssumptionThat the 2014 recovery went badly because storage servers stopped retrying the locking service and demoted themselves.

What happenedThe remediation was that "servers must continue to retry the locking service instead of giving up". Unlimited retries remove the cap on work amplification. In September 2015 the DynamoDB storage nodes "did not back out of retrying to get updated membership data".

Blast radiusTwo incidents, roughly fifteen months apart, in the same failure class at the same operator.

FixNot published as such. AWS's 2015 structural answer was segmenting the metadata service rather than revisiting the retry policy.

Design ruleEvery action item that increases persistence, retry count, logging detail or health-check aggressiveness strengthens a feedback loop. Ask of each one: what is the new worst-case work amplification, and where is the cap?

SourceHuang et al., OSDI '22, section 6.3

Paper

Spotify: the diagnostics that made the next incident worse

AssumptionThat better observability of the error path is free.

What happenedAfter a 2013 incident driven by load spikes and retries, engineers added significant logging to the error path to understand the cause. In the follow-up incident "the additional logging after a load spike and initial retries increased the cost of each retry, adding more load to the system and causing more requests to retry".

Blast radius8.33 hours for the second incident, against the core application and service UI.

FixTrigger hotfix plus load shedding.

Design ruleBudget the error path against the success path. If handling a failure costs more CPU, more IO or more network than handling a success, you have built an amplifier, and diagnostic logging is the most common way it gets built.

SourceHuang et al., OSDI '22, section 6.3

Eng blog

Meta link imbalance: two years, and the answer was in a connection pool

AssumptionThat traffic across an aggregated link is balanced by a hash the hosts cannot influence, so systematic imbalance is impossible.

What happenedQueries crossing a congested cable finish last, and a most-recently-used connection pool puts the last finisher on top of the stack. Each burst of cache misses therefore re-sorted the pool so that the slowest link was preferred next, concentrating traffic onto the congested cable.

Blast radiusMultiple outages over more than two years, from localised to site-wide. Root cause was declared prematurely several times, and switches from other vendors were tried.

FixA least-recently-used pool with a maximum connection age. Imbalance disappeared "within seconds of enabling the LRU policy".

Design ruleAny policy that prefers the resource that responded most recently is a positive feedback loop under contention. That includes MRU pools, least-response-time load balancing and cache eviction tuned on recency alone.

SourceMeta Engineering, 14 November 2014

Source

The safety mechanism that is configured and inert

AssumptionThat a retry throttle present in configuration is a throttle in effect.

What happenedIn grpc-java, the throttling policy is applied only when the service config comes from the name resolver. With a default service config and no resolver-supplied one, throttling is bypassed and every retry executes. Separately, Envoy's budget miscounts retries in backoff, so its effective limit differs from the configured one and varies by HTTP version.

Blast radiusUnknown and unmeasurable from public data, which is the point: this failure is invisible until an incident.

Fixgrpc-java: assigned to a release milestone in 2024. Envoy: the issue was closed as not planned and three pull requests to address it closed without merging between January 2024 and May 2026.

Design ruleAcceptance criteria for a retry budget is a metric, not a config line. Run a load test that exceeds it and show the rejection counter moving before you call the control implemented.

Sourcegrpc-java #11274, Envoy #30205, Envoy #43792

Numbers you can plan against

Everything quantitative in this guide, with the organisation, the context it was measured in, and the date. Read the note under the table before you use any of it in a capacity model.

Figure 6 · Advertised capacity is not the number that matters

The worked example from Bronson et al., HotOS '21, section 2.1: a database good for 300 QPS behind a client that retries once after one second. Everything between 150 and 300 QPS serves perfectly and cannot self-heal from a trigger. Almost all capacity planning targets the right-hand edge of the gold band.

Metric	Value	At	Context	As of	Source
Retry amplification, 5 layers, 3 attempts each	243×	Amazon	Stated as the arithmetic of independent per-layer retries	current	Builders' Library
Client-side amplification during recovery	~10×	GitHub	VS Code retry bug; Copilot Token Service 7 to 9K RPS becoming 70 to 100K RPS	2026-08	Incident summary
Worst-case work amplification with a priority system in place	>100×	Meta	Geo-distributed system with added retries and failover destinations	2021	HotOS '21
Query amplification from losing a 90% hit-rate cache	10×	Meta	Look-aside cache, worked example	2021	HotOS '21
Hidden capacity against advertised, retry example	150 / 300 QPS	Meta	One retry after a 1 s timeout; recovery needs load under 150 QPS	2021	HotOS '21
Hidden capacity against advertised, cache example	300 / 3,000 QPS	Meta	90% hit rate look-aside cache	2021	HotOS '21
Incidents whose sustaining effect was retry policy	>50%	11 organisations	22 classified metastable failures	2022-07	OSDI '22
Triggers that were engineer error rather than load	~45%	same sample	Buggy config or code deployment, and latent bugs; load spikes were about 35%	2022-07	OSDI '22
Incidents mitigated by load shedding	>50%	same sample	The most common mitigation of any kind	2022-07	OSDI '22
Outage duration range	1.5 to 73.5 h	same sample	4 to 10 hours is the mode, at 35% of incidents reporting a period	2022-07	OSDI '22
Major AWS outages of the decade in this class	≥4 of 15	AWS	Authors' classification of public incident reports	2022-07	OSDI '22
Envoy retry budget, default	20%	Envoy	Of active plus pending requests, floor of 3 concurrent retries	current	circuit_breaker.proto
Envoy fixed retry circuit breaker, default	3	Envoy	`max_retries`, the alternative to a budget	current	circuit_breaker.proto
gRPC retry throttle cut-off	maxTokens / 2	gRPC	Below this the client stops retrying and hedging to that server entirely	current	gRFC A6
Suggested per-process retry budget	60 per minute	Google	Given as an example figure, not a universal constant	2016	SRE book ch. 22
CoDel queue parameters	5 ms / 100 ms	Meta	Target queue delay and interval, "seems to work well"	2015-11	Fail at Scale
Goodput under a badly tuned shedder at 3,000 RPS	552 requests	Uber	About 40% of capacity, against near-capacity for the adaptive replacement	2023-11	Uber Engineering
Adaptive shedder overhead per request	~1 µs	Uber	Cinnamon, in production Go services	2023-11	Uber Engineering
Fleet restart rate during recovery	few hundred per hour	AWS Kinesis	Front-end servers rebuilding shard-maps contend with serving	2020-11	AWS summary
Fleet membership propagation delay	up to 1 h	AWS Kinesis	Time for an existing front-end member to learn of new participants	2020-11	AWS summary
Servers requested during the scaling response	1,200 in 14 min	Slack	07:01 to 07:15 PST; the provisioning path then failed	2021-01	Slack Engineering
Peak error rate	~55%	AWS DynamoDB	At 02:37 PDT, eighteen minutes after the trigger	2015-09	AWS summary
Shuffle sharding flow collision probability	~1 in 5.4 billion	Kubernetes	Two competing flows landing on the same queue set, typical parameters	current	KEP-1040
Call reduction from adding jitter	>50%	Amazon	Simulated contention workload with 100 contending clients	2015-03	AWS Architecture Blog

Read these carefully

Measured: the Uber goodput figures, the AWS incident timelines and error rates, the GitHub RPS figures, and the jitter simulation. Claimed but not independently checked: Uber's 1 microsecond overhead and the Kubernetes collision probability, both from the implementing team. Derived: the 243 times figure is 3 to the power of 5 and holds only when every layer retries independently and none of them budget; if one layer has a budget the product collapses toward that layer's cap. Unknown: nobody has published a before-and-after for introducing a retry budget, so there is no public figure for what a budget costs you in ordinary transient-fault availability. Envoy's 20% default has no published derivation that I could find. Treat it as a starting point to measure from, not as a researched value.

Two staleness notes. The Google SRE book figure of 60 retries per minute is from 2016 and is offered in the text as an illustration; do not port it. The Envoy defaults are current as of August 2026 but sit on top of the accounting behaviour described in section 3, so the effective budget is not the configured one.

How to work out your own hidden capacity

The number an architect actually needs is not in anyone's table, because it is a property of your system. Bronson and colleagues give a procedure, and it is cheap enough to run in a staging environment in an afternoon. Pick a characteristic metric: one that moves when the trigger arrives and only returns to normal once the failure resolves. Their production list is queueing delay, request latency, cache hit rate, page faults, timeout rates, thread counts, lock contention and connection counts. Queue delay is the best of them, because it is far less sensitive to workload mix than queries per second. Then run load at some level, apply a trigger that spikes the metric, remove the trigger, and watch whether the system quiesces without help. If it does, you are below hidden capacity at that load. Bisect.

The output is a single number: the load below which your system self-heals. Put it on the capacity dashboard next to the load you are actually running, and the gap between them is your exposure, expressed in the only unit that matters. Most teams will find they are living permanently on the wrong side of it, which is the point Bronson makes about efficiency: the vulnerable state is where the hardware budget lives. Knowing you are there is a different thing from being there by accident.

The evidence wall

Every source behind this page, graded, with what it shows and the one thing to carry out of it. Filter by kind. The full ledger, with the quote supporting each individual claim, ships beside this file as sources.md.

Paper Rockset, Penn State, UNH2021-05

Metastable Failures in Distributed Systems (HotOS '21)

The paper that named the pattern, written by engineers with a decade of Meta production behind them. Four worked case studies: request retries, look-aside cache, slow error handling, and link imbalance. Introduces hidden capacity, characteristic metrics and trigger intensity as operational concepts.

Carry forwardThe root cause is the sustaining loop, not the trigger, because many triggers reach the same state.

sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf

Paper Penn State, UNH, Twitter2022-07

Metastable Failures in the Wild (OSDI '22)

Twenty-two incidents from eleven organisations, classified from public postmortems, with a table giving trigger, sustaining effect and mitigation for each. Section 6.3, "Fix to Break", is the most useful two paragraphs in the entire literature for anyone who runs post-incident reviews.

Carry forwardRetries sustain more than half of these failures, and load shedding mitigates more than half.

usenix.org/system/files/osdi22-huang-lexiang.pdf

Postmortem AWS2015-09

Summary of the Amazon DynamoDB Service Disruption

The canonical example. A brief network disruption, membership payloads that had grown past the retrieval allowance because of global secondary index adoption, and retries that held the metadata service down. Recovery required pausing traffic because adding capacity needed the saturated service to answer.

Carry forwardA control plane whose administrative API shares a fate with its data path cannot be repaired under load.

aws.amazon.com/message/5467D2/

Postmortem AWS2025-10

AWS Service Event in Northern Virginia, 19 to 20 October 2025

Notable because AWS uses the term "congestive collapse" in its own prose, and because the collapse began after the triggering DNS defect was already fixed. EC2 launches remained impaired for roughly eleven hours past that point while DWFM re-established droplet leases it could not complete before timeout.

Carry forwardRate-limit recovery work by queue depth. AWS committed to exactly that as a corrective action.

aws.amazon.com/message/101925/

Postmortem AWS2021-12

AWS Service Event in Northern Virginia, 7 December 2021

An automated scaling activity produced a connection surge across the boundary between two internal networks. Retries turned congestion into persistent congestion, and a latent defect stopped the tested backoff behaviour from engaging. Monitoring data crossed the same congested path, so the operators were blind while diagnosing.

Carry forwardKeep the telemetry that diagnoses congestion off the path that congests.

aws.amazon.com/message/12721/

Postmortem AWS2020-11

Summary of the Amazon Kinesis Event in Northern Virginia

A small capacity addition pushed every front-end server past an operating system thread limit, leaving them with unusable shard-maps. The valuable content is the recovery arithmetic: a few hundred servers per hour, and up to an hour for fleet membership to propagate.

Carry forwardYour fleet's cold-start time is the floor on every outage in that system. Measure it once, deliberately.

aws.amazon.com/message/11201/

Postmortem GitHub2026-08

Incident summary for 17 August 2026

The official status thread carries the technical detail the blog post omits: the Istio sidecar concurrency limit, the four HAProxy nodes exhausting flow limits, the VS Code retry bug, the RPS figures, and the fact that pausing HAProxy on those nodes produced immediate broad recovery.

Carry forwardThe status thread is often a better engineering document than the blog post. Read both.

github.com/orgs/community/discussions/205164

Postmortem GitHub2026-08-20

The August 17 outage, and the work ahead

GitHub's Chief Technology Officer writing three days after a seven-hour outage. Short, and useful mainly for the remediation list, which is a commitment to build the layer this guide argues most stacks are missing.

Carry forwardThe 2026 fix is the 2016 prescription: retry limits, retry budgets and variable timeouts, applied consistently.

github.blog/news-insights/company-news/the-august-17-outage-and-the-work-ahead/

Postmortem Slack2021-02

Slack's Outage on January 4th 2021

Unusually candid about the scaling response becoming a second failure. Cold client caches on the first Monday back, an overloaded transit gateway, two autoscaling signals firing together, and a provisioning service that hit an operating system limit and a cloud quota while talking over the degraded network.

Carry forwardLoad-test the provisioning path itself, under the network conditions in which you will need it.

slack.engineering/slacks-outage-on-january-4th-2021/

Source Envoy2023-10

Issue #30205: circuit breaker retry budgets count retries inconsistently

The recommended mechanism's own bug report. Retries in backoff are counted against the limit but excluded from the denominator that sets it, so even a 100% budget rejects. The effective budget also varies by HTTP version. Closed as not planned.

Carry forwardVerify your budget with a rejection metric under load. Do not trust the configured percentage.

github.com/envoyproxy/envoy/issues/30205

Source Envoy2023-11 to 2024-01

PR #30738: retry budgets count scheduled retries towards budget

The first attempt to fix the accounting. Reviewers worried about breaking existing users and about the cost of the new gauges, one maintainer said the change warranted more discussion, and it was closed in January 2024 in favour of a redesign that also did not land.

Carry forwardCorrecting a resource-accounting bug is a behaviour change for everyone relying on the incorrect behaviour, which is why these stall.

github.com/envoyproxy/envoy/pull/30738

Source Envoy2026-03 to 2026-05

PR #43792: attempt admission control extension point

Closed unmerged by stalebot in May 2026. The thread is the interesting part: contributors disclose that their employers already run internal implementations for prioritised shedding of retries, while the upstream argument is about whether this deserves a specific extension point or a universal one for circuit breaking.

Carry forwardThe good version of this control exists privately at large operators. If you are not one, you are building on the public version with its known gap.

github.com/envoyproxy/envoy/pull/43792

Source gRPC2024-06

grpc-java #11274: retry throttling not respected with default service config

Throttling is wired up only when the service config arrives from the name resolver. With a default service config and no resolver-supplied one, the throttle is bypassed and every retry executes. A configured control that does nothing, which is the worst state for a safety mechanism to be in.

Carry forwardConfiguration provenance changes behaviour in client libraries. Test the control in the deployment shape you actually ship.

github.com/grpc/grpc-java/issues/11274

Source Netflixcurrent

Netflix/concurrency-limits

The library and its README, which is really a short design document. Equates a service's concurrency limit to a TCP congestion window, ships Vegas, Gradient2 and AIMD limiters, and is explicit that the server-side limiter exists to survive retry storms and batch traffic the service does not control.

Carry forwardDrive the limit from queue delay, not from CPU. Latency change is the signal that a queue is forming.

github.com/Netflix/concurrency-limits

Source Envoycurrent

circuit_breaker.proto API reference

The shipped defaults, which is what most deployments run: budget_percent at 20%, min_retry_concurrency at 3, and max_retries at 3 for the static alternative. Worth reading alongside issue #30205 to see what the percentage is a percentage of.

Carry forwardA budget is a ratio of live work; a max_retries is a count. Only the ratio scales with your traffic.

envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/circuit_breaker.proto

Decision record gRPCcurrent

gRFC A6: client retries

The design document for retries, hedging and throttling in every gRPC implementation. Specifies the per-server token bucket, the transparent retries that happen regardless of policy, the retryable status code rules, and server pushback via grpc-retry-pushback-ms, including the negative value that means do not retry at all.

Carry forwardGive servers a way to tell clients to stop. A pushback header is cheaper than every client guessing.

github.com/grpc/proposal/blob/master/A6-client-retries.md

Decision record Kubernetescurrent

KEP-1040: API Priority and Fairness

The argument for replacing a single global in-flight cap with priority levels, fair queuing and shuffle sharding, written as a design proposal with the failure scenarios spelled out: heartbeats crowded out, priority inversion via admission webhooks, one buggy controller taking down the cluster.

Carry forwardA single admission cap decides who loses by accident. Priority levels decide it in advance, in review.

github.com/kubernetes/enhancements KEP-1040

Eng blog Google2016

Site Reliability Engineering, chapter 22: Addressing Cascading Failures

The first widely read public statement of the retry budget and of the rule to retry at one level only. Also carries the recovery sequence that every incident in this guide eventually followed: reduce load until the crashing stops, let servers become healthy, ramp gradually.

Carry forwardAttempts multiply as the product across layers. That single sentence is the whole capacity argument.

sre.google/sre-book/addressing-cascading-failures/

Eng blog Meta2014-11

Solving the mystery of link imbalance

Nathan Bronson on a failure that took two years, several premature root causes, switches from a different vendor and a firmware feature before anyone found it. The cause was a most-recently-used connection pool preferring whichever network link was slowest. The fix was one line.

Carry forwardAny policy that prefers the most recently used resource is a positive feedback loop under contention.

engineering.fb.com/2014/11/14/production-engineering/solving-the-mystery-of-link-imbalance-a-metastable-failure-state-at-scale/

Eng blog Meta2015-11

Fail at Scale (Ben Maurer), as summarised by the morning paper

Meta's queue-side controls, published years before the failure class had a name: CoDel with a 5 ms target and a 100 ms interval, adaptive LIFO so that once a queue forms new requests go to the front and at least some meet their deadline, and per-service caps on outstanding client requests.

Carry forwardUnder overload, FIFO guarantees that everyone misses their deadline. LIFO guarantees that somebody does not.

blog.acolyer.org/2015/11/19/fail-at-scale-controlling-queue-delay/

Eng blog Uber2023-11

Cinnamon: using century-old tech to build a mean load shedder

The most quantitative countermeasure post in the set. Why hand-tuned per-service limits did not survive thousands of microservices, the measured goodput collapse of the old shedder at three times capacity, and a replacement built from a PID controller and a TCP-Vegas variant that needs no per-service configuration.

Carry forwardThe cost of a badly tuned shedder is measurable: 40% of capacity delivered, with oscillation between success and error.

uber.com/en-IN/blog/cinnamon-using-century-old-tech-to-build-a-mean-load-shedder/

Eng blog Marc Brooker2021-05

Metastability and Distributed Systems

A senior AWS engineer's reading of the HotOS paper, and the clearest short statement of why optimising the common case builds the loop. Frames the metastable state as "Up, but down. Working, but broken", which is the line that makes it recognisable on a dashboard where throughput looks fine.

Carry forwardWatch goodput, not throughput. A system in this state is busy, and busy is what your traffic graph shows.

brooker.co.za/blog/2021/05/24/metastable.html

Eng blog Amazon2015-03

Exponential Backoff And Jitter

The 2015 post that put full jitter and decorrelated jitter into every AWS SDK and most client libraries since. Worth reading for what it does and does not claim: it reduces contention and call counts, and it says nothing about bounding total work.

Carry forwardJitter decorrelates a herd. It does not cap one. Do not let it stand in for a budget in a design review.

aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/

Eng blog Grab2019-03

Designing Resilient Systems Beyond Retries, part 1

A ride-hailing platform's argument for server-side rate limiting as the primary defence, on the grounds that client-side controls can fail or be bypassed and that coordinating circuit-breaker configuration across hundreds of microservices is not sustainable.

Carry forwardA server that depends on its clients for protection has no protection. Client controls are an optimisation on top.

engineering.grab.com/beyond-retries-part-1

Eng blog Cloudflare2026

Code Orange: Fail Small

Included as a contrast case rather than a retry story. Cloudflare's global incidents of November and December 2025 propagated through configuration rather than load, and the structural response is staged rollout with health mediation plus validated defaults that fail open.

Carry forwardBlast radius and feedback loops are different problems. Cellular isolation limits one and does nothing for the other.

blog.cloudflare.com/fail-small-resilience-plan/

Vendor Amazonretrieved 2026-08

Timeouts, retries, and backoff with jitter (Builders' Library)

Marc Brooker writing AWS practice rather than AWS marketing, which puts it well above the usual vendor tier. Carries the 243 times arithmetic, the argument that retries are selfish, the explicit rejection of circuit breakers in favour of token buckets, and the note that this went into the SDKs in 2016.

Carry forwardRetry at a single point in the stack, and use consistent per-host jitter on scheduled work so that patterns stay diagnosable.

d1.awsstatic.com · timeouts-retries-and-backoff-with-jitter.pdf

Vendor Envoycurrent

Circuit breaking, architecture overview

Short, and useful for one sentence: the project recommends retry budgets over static circuit breaking, and says that if you insist on static breaking you should break retries aggressively. That is a proxy vendor telling you its own default is the weaker option.

Carry forwardIf you keep max_retries, set it low and alarm on the overflow counter, which is the URX response flag.

envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/circuit_breaking

Talk USENIX OSDI '222022-07-11

Metastable Failures in the Wild, conference presentation

The conference record for the survey paper, with slides and presentation video attached alongside the PDF. Useful if you need to put the incident table in front of a review board and want the citation rather than the blog summary.

Carry forwardCite the primary. Several widely shared summaries of this paper get the sample size and the AWS statistic wrong.

usenix.org/conference/osdi22/presentation/huang-lexiang

Talk SREcon23 APAC2023-06-16

Mastering Chaos: observability-driven prioritized load shedding

A vendor-affiliated talk, so weigh the product claims accordingly, but slides 3 to 10 are the best short visual explanation of the loop available. Slide 4 frames it through Little's law, slide 8 shows retries increasing pressure on the database, and slide 9 shows permanent overload persisting after capacity is restored.

Carry forwardLittle's law is the framing that makes concurrency the controlled variable rather than requests per second.

usenix.org/conference/srecon23apac/presentation/gill

Vendor USENIX2023-06-16

SREcon23 APAC slide deck (PDF)

The deck for the talk above, cited here because the specific slides are quotable and the video is not. Thirty-six slides; the material relevant to this guide is in the first eleven, before the product content begins.

Carry forwardWhen citing a talk, cite the deck. Slide numbers are checkable and timestamps in a video are not.

usenix.org/system/files/srecon23apac_slides-gill_0.pdf

Where the record runs out

Four gaps, named because they are where your risk sits. First, no organisation has published a measured before-and-after for introducing a retry budget: what the amplification factor was, what it became, and what it cost in availability during ordinary transient faults. Second, no public account describes deadline propagation and retry budgets interacting, although the same sources recommend both. Third, nobody has priced the headroom. Running in the stable rather than the vulnerable state is a hardware purchase and no public source says what it costs, which is precisely why teams choose the vulnerable state. Fourth, the OSDI '22 sample is 22 incidents that were public and detailed enough to classify, which makes it a sample of what large companies chose to write down rather than of what happens.

Build a miniature, then productionise it

Six rungs. The first three fit in an evening on a laptop, and the crossing into production shape happens at rung four, where the thing you are building stops being a demo and starts being a measurement.

Reproduce the collapse in fifty lines

A client, a server with a fixed worker pool and an artificial service time, and a client timeout with one retry. Drive it at 90% of the server's capacity, drop a two-second outage in the middle, and remove it.

Done when: the offered load returns to its original value and goodput stays near zero for at least a minute. Teaches: that a system can be fully loaded and delivering nothing, which is the thing dashboards hide.

Find the hidden capacity of your toy

Bisect on offered load. At each level apply the same trigger and record whether the system quiesces by itself. Plot goodput against load and mark the two thresholds.

Done when: you can state a load below which the trigger is harmless and above which it is fatal, and the two numbers differ by roughly the amplification factor. Teaches: that advertised capacity and recoverable capacity are different numbers, and only one is on your dashboard.

Add backoff and jitter, and watch it not help

Add capped exponential backoff with full jitter to the client, then repeat rung two. Compare the two hidden-capacity numbers.

Done when: you can show that the collapse still happens, at a slightly higher load and with a smoother arrival pattern. Teaches: the difference between a scheduling policy and a bound, which is the most common confusion in a design review.

Add a token bucket, and measure what it costs

Implement the gRPC scheme: start at maxTokens, decrement on failure, increment by tokenRatio on success, stop retrying below half. Then measure the thing nobody publishes: run a workload with a 1% random transient error rate and compare end-user success rate with and without the bucket.

Done when: you have two numbers, the hidden capacity gain and the transient-fault availability loss, and can defend the trade. Teaches: that a budget is a purchase, and that you are the first person in your organisation to know its price.

Move the control to the server and shed by priority

Replace the client bucket with server-side admission: an adaptive concurrency limit driven by queue delay rather than CPU, returning 429 with a pushback hint. Tag half the traffic as low priority and shed it first.

Done when: under three times capacity, high-priority goodput stays within 20% of nominal while low-priority is rejected quickly rather than timing out. Teaches: that shedding early and cheaply beats queueing, because a request that times out after five seconds consumed five seconds.

Run the drill against your real system, on a weekday

Pick your highest-traffic internal dependency. In a load environment, induce thirty seconds of added latency, remove it, and time how long goodput takes to recover. Then rehearse the load-shedding lever: find it, confirm someone has permission to pull it, and pull it.

Done when: recovery time is a number in the runbook and the shedding lever has been used at least once by someone who was not its author. Teaches: the lever every incident in section 4 needed, which in most organisations exists on a slide and not in production.

Keep hunting

The queries that found the material above. The vocabulary matters more than the operators: "congestive collapse" and "sustaining effect" are inside-the-field terms that return incident reports, while "retry best practices" returns tutorials.

Incident reports, in the operators' own words

"congestive collapse" postmortem site:aws.amazon.com
"continued to retry" OR "did not back off" incident summary
"we paused" OR "we throttled" recovery outage summary
"even after" "was restored" outage "did not recover"

The research vocabulary

"metastable failure" "sustaining effect" -tutorial
"hidden capacity" "characteristic metric" distributed systems
"work amplification" retry goodput site:usenix.org
"Bronson" metastable cited by

The argument, in issue trackers

repo:envoyproxy/envoy is:pr is:closed is:unmerged retry budget
"retry budget" OR "retry throttling" is:issue label:design
path:docs/adr retry OR "load shedding" OR "admission control"
"adaptive concurrency" OR "congestion window" language:go stars:>500

Countermeasures with numbers in them

intitle:"how we" load shedder OR "overload control" goodput
"we replaced" circuit breaker "retry budget" engineering blog
CoDel OR "adaptive LIFO" production queue delay -tutorial
"load shedding" site:usenix.org srecon slides

References

Bronson, Aghayev, Charapko, Zhu. Metastable Failures in Distributed Systems HotOS '21, ACM, May 2021. Checked 2026-08-29.
Huang, Magnusson, Bangalore Muralikrishna, Estyak, Isaacs, Aghayev, Zhu, Charapko. Metastable Failures in the Wild OSDI '22, USENIX, July 2022, pages 73 to 90. Checked 2026-08-29.
Metastable Failures in the Wild, conference presentation page USENIX, July 2022. Slides and video. Checked 2026-08-29.
Summary of the Amazon DynamoDB Service Disruption and Related Impacts in the US-East Region Amazon Web Services, September 2015. Checked 2026-08-29.
Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region Amazon Web Services, November 2020. Checked 2026-08-29.
Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region Amazon Web Services, December 2021. Checked 2026-08-29.
Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region, October 2025 Amazon Web Services, October 2025. Checked 2026-08-29.
Slack's Outage on January 4th 2021 Slack Engineering, February 2021. Checked 2026-08-29.
[2026-08-17] Incident Thread, including the official incident summary GitHub Community, August 2026. Checked 2026-08-29.
Vlad Fedorov. The August 17 outage, and the work ahead The GitHub Blog, 20 August 2026. Checked 2026-08-29.
Site Reliability Engineering, chapter 22: Addressing Cascading Failures Google, O'Reilly, 2016. Checked 2026-08-29.
Marc Brooker. Timeouts, retries, and backoff with jitter Amazon Builders' Library, undated, retrieved 2026-08-29.
Marc Brooker. Exponential Backoff And Jitter AWS Architecture Blog, 4 March 2015. Checked 2026-08-29.
Marc Brooker. Metastability and Distributed Systems Marc's Blog, 24 May 2021. Checked 2026-08-29.
Nathan Bronson. Solving the mystery of link imbalance: a metastable failure state at scale Meta Engineering, 14 November 2014. Checked 2026-08-29.
Fail at Scale and Controlling Queue Delay the morning paper, 19 November 2015, summarising Ben Maurer, ACM Queue 13(8). Checked 2026-08-29.
Smyth, Gavrilenko, Lindstrom Nielsen, Holdgaard Thomsen. Cinnamon: using century-old tech to build a mean load shedder Uber Engineering, 22 November 2023. Checked 2026-08-29.
Michael Cartmell. Designing Resilient Systems Beyond Retries, Part 1: Rate-Limiting Grab Engineering, 20 March 2019. Checked 2026-08-29.
Code Orange: Fail Small, our resilience plan following recent incidents The Cloudflare Blog, 2026. Checked 2026-08-29.
gRFC A6: gRPC Retry Design grpc/proposal, current. Checked 2026-08-29.
KEP-1040: Priority and Fairness for API Server Requests kubernetes/enhancements, current. Checked 2026-08-29.
Envoy circuit_breaker.proto API reference Envoy Proxy documentation, current. Checked 2026-08-29.
Envoy architecture overview: circuit breaking Envoy Proxy documentation, current. Checked 2026-08-29.
Envoy issue #30205: circuit breaker retry budgets count retries inconsistently Opened 13 October 2023, closed as not planned. Checked 2026-08-29.
Envoy PR #30738: circuit breakers, retry budgets count scheduled retries towards budget Opened 6 November 2023, closed unmerged 2 January 2024. Checked 2026-08-29.
Envoy PR #43792: attempt admission control extension point for circuit breaker Opened 5 March 2026, closed unmerged 21 May 2026. Checked 2026-08-29.
grpc-java issue #11274: retry throttling not being respected with default service config Opened 10 June 2024. Checked 2026-08-29.
Netflix concurrency-limits Netflix, current. Checked 2026-08-29.
Harjot Gill, Hardik Shingala. Mastering Chaos: Achieving Fault Tolerance with Observability-Driven Prioritized Load Shedding SREcon23 APAC, USENIX, Singapore, 16 June 2023. Checked 2026-08-29.
Mastering Chaos, slide deck USENIX, June 2023, 36 slides. Checked 2026-08-29.