Evidence ledger
28 sources
Checked 29 Aug 2026
Evidence ledger
One row per claim in Keeping one failure from reaching everyone: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how production systems partition themselves so one failure reaches a fraction of users, and why the containment keeps leaking through whatever stayed global.
Research date: 2026-08-29. Every URL below was fetched during that session; the Checked
column records it. One row per claim. Quotes are copied, not paraphrased.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | AWS | Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region | postmortem | 2025-10 | 2026-08-29 | https://aws.amazon.com/message/101925/ | The routing layer, not a cell, was the single point of failure: an empty DNS record for the regional endpoint made every cell behind it unreachable. | "a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record" for dynamodb.us-east-1.amazonaws.com. |
| 2 | AWS | Same | postmortem | 2025-10 | 2026-08-29 | https://aws.amazon.com/message/101925/ | Two redundant actors operating on shared global state produced the failure; redundancy in the change path is not isolation. | The DNS Enactor runs "redundantly across three Availability Zones"; a delayed Enactor applied "a much older plan", overwriting the newer one, and cleanup then deleted it, "removing all IP addresses for the regional endpoint". |
| 3 | AWS | Same | postmortem | 2025-10 | 2026-08-29 | https://aws.amazon.com/message/101925/ | Recovery, not the fault, produced the long tail: EC2's lease manager could not converge once the dependency returned. | "DWFM had entered a state of congestive collapse and was unable to make forward progress." DynamoDB DNS restored 02:25 PDT; EC2 throttling lifted 13:50 PDT. |
| 4 | Cloudflare | Cloudflare outage on November 18, 2025 | postmortem | 2025-11-18 | 2026-08-29 | https://blog.cloudflare.com/18-november-2025-outage/ | A config artefact regenerated every few minutes and pushed to the whole fleet is a global change plane, whatever the request path looks like. | The Bot Management feature file "is refreshed every few minutes and published to the entire network"; a ClickHouse permissions change made the generating query return duplicate rows because "the query does not filter for the database name". |
| 5 | Cloudflare | Same | postmortem | 2025-11-18 | 2026-08-29 | https://blog.cloudflare.com/18-november-2025-outage/ | A hard limit plus an unchecked unwrap turns oversized input into a fleet-wide crash. | "the Bot Management system has a limit on the number of machine learning features that can be used at runtime. Currently that limit is set to 200"; the proxy "called Result::unwrap() on an Err value". |
| 6 | Cloudflare | Same | postmortem | 2025-11-18 | 2026-08-29 | https://blog.cloudflare.com/18-november-2025-outage/ | Gradual rollout of the upstream change made the failure intermittent, which delayed diagnosis and mimicked an attack. | "Every five minutes there was a chance of either a good or a bad set of configuration files being generated." Impact 11:20 UTC, core traffic normal 14:30, all systems 17:06. |
| 7 | Datadog | 2023-03-08 Incident: Infrastructure connectivity issue affecting multiple regions | postmortem | 2023-03 | 2026-08-29 | https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/ | Five regions on three clouds failed simultaneously because they shared an OS update channel and a default update window, not a network. | "The time at which the automatic update happens is set by default in the OS to a window between 06:00 and 07:00 UTC. Thus it affected multiple regions at the exact same time, even though the regions have no direct network connection or coupling between them." |
| 8 | Datadog | Same | postmortem | 2023-03 | 2026-08-29 | https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/ | The mechanism was a latent interaction only reachable on a running node, so no boot-time test would find it. | "systemd-networkd forcibly deleted the routes managed by the Container Network Interface (CNI) plugin (Cilium)"; "Neither a fresh node nor a rebooted node exhibit this behavior". Affected "tens of thousands of nodes in our fleet between 06:00 and 07:00". |
| 9 | The Pragmatic Engineer | Inside Datadog's $5M Outage | blog | 2023-05-16 | 2026-08-29 | https://newsletter.pragmaticengineer.com/p/inside-the-datadog-outage | Independent estimate of the cost and the size of the response, and confirmation that the shared artefact was the OS image. | "$5 million" in lost revenue, "about a day's worth of revenue"; "Between 500 - 600 engineers working tirelessly"; same OS image used across cloud providers. |
| 10 | Roblox | Roblox Return to Service 10/28-10/31 2021 | postmortem | 2022-01 | 2026-08-29 | https://about.roblox.com/newsroom/2022/01/roblox-return-to-service-10-28-10-31-2021 | One shared coordination cluster is the whole blast radius; the fix is stated as partitioning it. | "A single Consul cluster supporting multiple workloads exacerbated the impact of these issues." 73-hour outage, 28-31 October 2021. |
| 11 | Roblox | Same | postmortem | 2022-01 | 2026-08-29 | https://about.roblox.com/newsroom/2022/01/roblox-return-to-service-10-28-10-31-2021 | Observability inside the failure domain extends the outage, because the detector dies with the thing it watches. | "There was a circular dependency between our telemetry systems and Consul, which meant that when Consul was unhealthy, we lacked the telemetry data that would have made it easier for us to figure out what was wrong." |
| 12 | CrowdStrike | Channel File 291 Incident Root Cause Analysis | postmortem | 2024-08-06 | 2026-08-29 | https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf | Content shipped through a channel deliberately built to bypass staged release will reach every host at once. | "The new IPC Template Type defined 21 input parameter fields, but the integration code that invoked the Content Interpreter... only supplied 20." Mitigation 6: "Template Instances should have staged deployment." |
| 13 | CrowdStrike | Same | postmortem | 2024-08-06 | 2026-08-29 | https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf | The remediation is a ring topology on the change path, plus giving the customer control of their own ring. | "Template Instances that have passed canary testing are to be successively promoted to wider deployment rings or rolled back if problems are detected." Plus: "Provide customer control over the deployment of Rapid Response Content." |
| 14 | Amazon / USENIX | Millions of Tiny Databases (Physalia), NSDI '20 | paper | 2020-02 | 2026-08-29 | https://www.usenix.org/conference/nsdi20/presentation/brooker | Cells are the primary blast-radius tool, and independence is bought deliberately at the cost of a separate control plane. | §2.1: "The division of a colony into a large number of cells is our main tool for reducing radius in Physalia." "We chose to keep our cells completely independent, and implement the control plane as a seperate system." |
| 15 | Amazon / USENIX | Same | paper | 2020-02 | 2026-08-29 | https://cdn.amazon.science/c4/11/de2606884b63bf4d95190a3c2390/millions-of-tiny-databases.pdf | Cell size is a trade-off between random and correlated failure, not a free dial; EBS's Physalia uses seven nodes. | §2.2: "the cell performs Paxos over seven nodes"; "smaller cells offer lower availability in the face of small numbers of uncorrelated node failures, but better availability when the proportion of node failure exceeds 50%." |
| 16 | Amazon / USENIX | Same | paper | 2020-02 | 2026-08-29 | https://cdn.amazon.science/c4/11/de2606884b63bf4d95190a3c2390/millions-of-tiny-databases.pdf | The paper names the failure class cells do not contain, and introduces a second, orthogonal partition (colors) for the change path. | §3.4: "Each cell is assigned a color, and each cell is constructed only of nodes of the same color... When software deployments and other operations are performed, they proceed color-by-color." "Nodes of different colors don't communicate with each other, making it significantly less likely that a poison pill or overload could spread across colors." |
| 17 | Amazon / USENIX | Same | paper | 2020-02 | 2026-08-29 | https://cdn.amazon.science/c4/11/de2606884b63bf4d95190a3c2390/millions-of-tiny-databases.pdf | Replication makes staged rollout weaker, not stronger, because a fault-tolerant system hides the canary signal. | §3.3: "The fault-tolerant nature of distributed state machines makes this approach less effective: because the system is designed to tolerate failure in less than half of hosts, failure may not be evident until new code is deployed to half of all hosts." |
| 18 | Amazon / USENIX | Same | paper | 2020-02 | 2026-08-29 | https://cdn.amazon.science/c4/11/de2606884b63bf4d95190a3c2390/millions-of-tiny-databases.pdf | Operations, not hardware, dominate downtime; this is the stated reason colors exist. | §3.4: "operations, including code and configuration deployments, routine system operations such as security patching, and scaling for increased load, are dominant contributors to system downtime". |
| 19 | Microsoft Research | Gray Failure: The Achilles' Heel of Cloud-Scale Systems, HotOS '17 | paper | 2017-05 | 2026-08-29 | https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/paper-1.pdf | The formal definition of the failure mode that makes automatic containment fail: the detector and the caller disagree. | §3.2: "a system is defined to experience gray failure when at least one app makes the observation that system is unhealthy, but observer observes that system is healthy." |
| 20 | Microsoft Research | Same | paper | 2017-05 | 2026-08-29 | https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/paper-1.pdf | Under fan-out, redundancy raises the probability that a request touches the degraded component. This is why one bad AZ degrades everything. | §2.1 "High redundancy hurts": with n core switches and fan-out m, the chance a request traverses a given switch is 1-((n-1)/n)^m, which "rapidly approaches 100% as m becomes large... increasing redundancy can counter-intuitively hurt availability". |
| 21 | AWS | ARC411-R1: Reducing blast radius with cell-based architectures, re:Invent 2019 | talk | 2019-12 | 2026-08-29 | https://d1.awsstatic.com/events/reinvent/2019/REPEAT_1_Reducing_blast_radius_with_cell-based_architectures_ARC411-R1.pdf | The canonical cell definition and the router rule, from AWS's own practice. | Slide "What is a cell?": "Fixed amount of resources / Well-defined function / No calling other cells / Will not expand / Diverse". Slide "Router properties": "Keep it simple!", "Thinnest possible layer". |
| 22 | AWS | Same | talk | 2019-12 | 2026-08-29 | https://d1.awsstatic.com/events/reinvent/2019/REPEAT_1_Reducing_blast_radius_with_cell-based_architectures_ARC411-R1.pdf | Small versus large cells is the sizing decision, and the trade is blast radius against cost and operational load. | Slide 9: smaller cells give "Reduced blast radius / Easier to test / Cells easier to operate"; larger cells give "Cost efficiency / Reduced splits / System easier to operate". |
| 23 | Slack / InfoQ | Slack's Migration to a Cellular Architecture (Cooper Bethea, QCon SF 2024) | talk | 2025-05-02 | 2026-08-29 | https://www.infoq.com/presentations/slack-cellular-architecture/ | The migration's binding constraint was money and organisational coordination, not design. | "If we're paying double for the site the whole time, we're going to get broke." Drain time of 60 seconds for siloed services; some critical services took "a year, or more than a year". |
| 24 | Slack | Slack's Migration to a Cellular Architecture | blog | 2023-08 | 2026-08-29 | https://slack.engineering/slacks-migration-to-a-cellular-architecture/ | The trigger was a partial AZ fault that no health check caught, on 30 June 2021. | "A network link that connects one availability zone with several other availability zones containing Slack servers experienced intermittent faults"; "systems within the impacted AZ saw complete availability of backends within their AZ, but backends outside the AZ were unavailable, and vice versa". |
| 25 | Slack | Same | blog | 2023-08 | 2026-08-29 | https://slack.engineering/slacks-migration-to-a-cellular-architecture/ | The design is siloing plus a fast, granular drain, and the drain is the product, not the cells. | "All services are present in all AZs, but each service only communicates with services within its AZ." "Propagation through the control plane is on the order of seconds"; "Weights provide gradual drains with a granularity of 1%." |
| 26 | Slack | Same | blog | 2023-08 | 2026-08-29 | https://slack.engineering/slacks-migration-to-a-cellular-architecture/ | Legacy discovery interfaces are the practical obstacle to draining, because DNS cannot express a partial drain. | "DNS does not offer an abstraction for something like an AZ or partial draining." |
| 27 | Shopify | A Pods Architecture To Allow Shopify To Scale | blog | 2019 | 2026-08-29 | https://shopify.engineering/a-pods-architecture-to-allow-shopify-to-scale | Sharding alone does not isolate; the isolation came from making a request need exactly one shard. | "A pod consists of a set of shops that live on a fully isolated set of datastores." Before pods: "If any of our shards went down, that entire action would be unavailable across the platform." After: "serving a request only requires a single pod to be online". |
| 28 | Salesforce | Architectural Principles for High Availability on Hyperforce | blog | 2022-08-10 | 2026-08-29 | https://engineering.salesforce.com/architectural-principles-for-high-availability-on-hyperforce/ | Cells and AZs are composed, not substituted: a cell spans AZs, and cells stagger deployment. | "A cell is a collection of services serving a group of our customers... deployed across multiple availability zones for redundancy within the cell and is isolated from other cells." Deployment: "canary on a small set of cells, and then stagger the deployment across other cells." |
| 29 | InfoQ / DoorDash | DoorDash Uses Service Mesh and Cell-Based Architecture to Significantly Reduce Data Transfer Costs | blog | 2024-01-23 | 2026-08-29 | https://www.infoq.com/news/2024/01/doordash-service-mesh/ | The motive for cells here was cross-AZ data-transfer cost, and resilience was the by-product. | Each microservice is "deployed exclusively to one cluster within a given cell"; zone-aware routing via Envoy; the saving "caused our cloud provider to reach out to us asking whether we were experiencing a production-related incident". |
| 30 | InfoQ / DoorDash | Same | blog | 2024-01-23 | 2026-08-29 | https://www.infoq.com/news/2024/01/doordash-service-mesh/ | Zone affinity has to hold across every hop, and the visibility to prove it does not exist off the shelf. | "As microservice call graph hops increase, ensuring all hops support zone-aware routing becomes more complex"; "It's challenging to build a comprehensive understanding/view of all cross-AZ traffic." |
| 31 | Kubernetes | KEP-2433: Topology Aware Hints | adr | 2021 | 2026-08-29 | https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/2433-topology-aware-hints/README.md | The platform's zone affinity is explicitly a preference built on an assumption about traffic, and it disables itself. | Assumes "Incoming traffic is proportional to the number of allocatable CPU cores in a zone". "Hints will not be added for a Service unless the expected initial overload is below 20%"; "if any endpoints for a Service do not have a hint, kube-proxy will ignore all hints." |
| 32 | Kubernetes | KEP-4444: Traffic Distribution for Services | adr | 2024-2026 | 2026-08-29 | https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/4444-service-traffic-distribution/README.md | The heuristic approach was superseded because it was unpredictable in practice; users could not tell whether it was on. | "While designed for intelligent routing, this mode offers less user control and can be less predictable. Users filed issues reporting that hints weren't being applied or didn't function as expected." Alpha 1.30, beta 1.31, GA 1.33. |
| 33 | Kubernetes | Topology Aware Routing (docs) | vendor | 2026 | 2026-08-29 | https://kubernetes.io/docs/concepts/services-networking/topology-aware-routing | The five documented conditions under which zone affinity silently stops applying. | "If any node does not have a topology.kubernetes.io/zone label or is not reporting a value for allocatable CPU, the control plane does not set any topology-aware endpoint hints"; "This approach will not work well for Services that have a large proportion of traffic originating from a subset of zones." |
| 34 | Kubernetes | core/v1: deprecate Service topologyKeys (PR #96736) | source | 2020-12-09 | 2026-08-29 | https://github.com/kubernetes/kubernetes/pull/96736 | The explicit, user-specified topology API was rejected; the replacement is automatic and advisory. | "Deprecate the topologyKeys field in Service. This capability will be replaced with upcoming work around Topology Aware Subsetting and Service Internal Traffic Policy." Deprecated 1.21, removed 1.22. |
| 35 | Envoy | Issue #28419: Zone-aware load balancing incorrectly handles mismatched local and upstream localities | source | 2023-07-16 | 2026-08-29 | https://github.com/envoyproxy/envoy/issues/28419 | Zone affinity in the mesh was wrong whenever zone membership was asymmetric, which is the normal case. | Three bugs reported, including that sorting "collapses empty localities, misaligning locality-to-index mappings" so that "despite zones C and D being interchangeable" one receives more traffic. Fixed via PR #28970 (Envoy 1.28.0). |
| 36 | Envoy | Issue #43665: Proposal, ORCA-driven locality routing | source | 2026-02-26 | 2026-08-29 | https://github.com/envoyproxy/envoy/issues/43665 | Zone-aware routing still routes on host counts, not load, so uneven demand produces hotspots inside a zone. | "Zone-aware routing decides what percentage of traffic to keep local vs remote based on healthy host counts (default) or host weights. It has no visibility into actual backend load." |
| 37 | Istio | Issue #18500: support zone aware routing of envoy | source | 2019-10-31 | 2026-08-29 | https://github.com/istio/istio/issues/18500 | A six-year-old request that never landed as Envoy's zone-aware routing; the mesh offers locality weights instead, which imbalance under unequal replicas. | Locality weighted load balancing "may causes imbalance across all upstream hosts if replicas of a service is NOT equal in each zones." |
| 38 | AWS | guidance-for-cell-based-architecture-on-aws (reference implementation) | source | 2023-2026 | 2026-08-29 | https://github.com/aws-solutions-library-samples/guidance-for-cell-based-architecture-on-aws | Even the vendor's own reference pipeline canaries one cell then updates the rest together. | The pipeline "first updates the sandbox cell, checks it for aliveness and then updates all other cells." |
| 39 | AWS | Well-Architected: Reducing the Scope of Impact with Cell-Based Architecture | vendor | 2023-09-20 | 2026-08-29 | https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/reducing-scope-of-impact-with-cell-based-architecture.html | AWS positions cells as the workload-level version of the AZ and Region isolation boundaries. | "bringing the same fault isolation concepts that AWS applies in its Availability Zones and Regions to the level of your workload architecture". Published 20 September 2023. |
| 40 | AWS | Well-Architected: Cell design | vendor | 2023-09 | 2026-08-29 | https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/cell-design.html | The stated ideal is total independence, and the stated reality is that cross-cell dependencies destroy the benefit. | "Cells should have no dependency on each other at all (that is, no cross-cell API calls, no shared resources like databases or S3 buckets.)... Cross-cell dependencies can quickly eliminate the benefits of a cellular architecture". |
| 41 | AWS | Well-Architected: Cell partition | vendor | 2023-09 | 2026-08-29 | https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/cell-partition.html | The partition key must match the grain of the workload, and a single large tenant breaks a naive customer-ID key. | "Partition keys must be chosen to match the grain of the service"; on CustomerID: "a single customer of yours becomes so big that it doesn't fit into a single cell anymore". Cross-cell calls should "go back through the normal cell router". |
| 42 | AWS | Well-Architected: Cell deployment | vendor | 2023-09 | 2026-08-29 | https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/cell-deployment.html | AWS states the wave rule explicitly, and states the operational cost of cells honestly. | "deploy in waves, cell by cell or set of cells"; "now you have tens, hundreds or even thousands of instances of your workload to deploy and operate... it is a very complex challenge to deploy in a production environment." |
| 43 | AWS | Overview of Data Transfer Costs for Common Architectures | vendor | 2021-06-30 | 2026-08-29 | https://aws.amazon.com/blogs/architecture/overview-of-data-transfer-costs-for-common-architectures/ | Same-AZ traffic is free and cross-AZ traffic is charged, which is the economic force behind AZ-shaped cells. | "Data transfer within the same Availability Zone is free"; cross-AZ communication incurs data transfer charges. |
| 44 | Kubernetes | Issue #113731: Support topology aware routing (node and zone) | source | 2022-11-08 | 2026-08-29 | https://github.com/kubernetes/kubernetes/issues/113731 | The community's own statement of why the hints heuristic is not a boundary: it is non-deterministic and can disable itself wrongly. | "Sometimes the CPU-balance heuristic is wrong. For example, if we know that the whole zonal CPU pool does not model the set of clients which will access a service, then we may erroneously disable hints." Also: "Hints do not work at low-endpoint counts"; "Hints are not deterministic." |
Tier mix
| Tier | Rows | Distinct artefacts |
|---|---|---|
| postmortem | 13 | 5 |
| paper | 7 | 2 |
| talk | 3 | 2 |
| blog | 9 | 6 |
| source | 6 | 6 |
| adr | 2 | 2 |
| vendor | 6 | 5 |
| Total | 44 claim rows | 28 artefacts across 17 hosts |
Organisations represented: AWS/Amazon, Cloudflare, Datadog, Roblox, CrowdStrike, Slack, Shopify, Salesforce, DoorDash, Microsoft Research, Kubernetes (SIG Network), Envoy, Istio, The Pragmatic Engineer, InfoQ, USENIX.
What is missing, and it matters
- No published postmortem in this corpus describes a single cell failing and the failure staying inside it. That is either the strongest possible evidence that cells work, or evidence that a contained failure is never newsworthy enough to write up. Both readings argue the same thing: you will not learn your containment works from other people's blogs, only from your own drills.
- No public account gives a per-cell cost multiplier. Slack says the double-running cost was prohibitive; nobody publishes the number. Treat any figure you are quoted as unverified.
- DoorDash's own engineering post could not be retrieved during this session (HTTP 403 from
careersatdoordash.com); the DoorDash claims here are cited to InfoQ's January 2024 report, which quotes their engineers directly. Gradedblog, and flagged as second-hand in the page.