Keeping one failure from reaching everyone
How production systems partition themselves so one failure reaches a fraction of users, and why the containment keeps leaking through whatever stayed global.
Slack, Shopify, Salesforce, DoorDash, Amazon EBS and AWS all cut their systems into independent replicas so that one fault hurts some customers rather than all of them, and all of them publish how. This guide reconstructs the common architecture across those six, then shows from five public postmortems why the pattern keeps failing anyway: the partitioning covers the request path, while the faults arrive through the change path, a plane that is almost always still global. A reader finishes with a decision table for cell boundary, size and routing, a six-rung build ladder that crosses from cells to change-plane waves, and the specific questions to ask of their own pipeline on Monday.
The 2020 Physalia paper had already identified that cells do not contain deployments and had built a second, orthogonal partition for it, called colours, that operates on the change path rather than the request path; every large outage since is a failure of exactly that second partition, and none of the four organisations involved had one.
What you get out of it
- Cells partition the request plane; the change plane, which carries config, content, DNS, packages and code, is a second axis that almost nobody partitions, and it is where every large recent outage propagated.
- No published postmortem in this corpus describes a single cell failing and the failure staying inside it, which means the public record is structurally incapable of telling you whether your containment works; only a production drain can.
- The drain, not the cell, is the thing an incident commander uses. Slack targeted five minutes and achieved 60 seconds with 1% weight granularity; without a fast drain, cells buy a smaller incident rather than a shorter one.
- Kubernetes zone affinity is a cost preference, not an isolation boundary. It silently withholds hints when any node lacks a zone label or allocatable CPU, and its own successor KEP calls the heuristic unpredictable.
- AWS's published reference implementation deploys to a sandbox cell, checks aliveness, then updates all other cells, which is the exact change-plane anti-pattern its own Well-Architected guidance warns against.
Scope
Why this, now. Four of the five largest published outages of the last three years, Cloudflare and AWS in late 2025, CrowdStrike in 2024 and Datadog in 2023, propagated through unpartitioned change machinery rather than through anything cellular, making this the failure class an architect is most likely to meet next.
What it does not cover. Multi-region active-active failover, sharding purely for scale, single-tenant per-customer deployments, the security case for tenant isolation, and Kubernetes multi-cluster management tooling.
Other field guides
The cutover is the easy part
A field guide to online resharding, reconstructed from thirteen production accounts (Notion, Figma, Slack, GitHub, Shopify, Discord, Etsy, Pinterest,…
34 sources · 23 organisations · 5 postmortemsThe renewal failed a month before anyone noticed
A field guide to the machine credential lifecycle, built from ten published incidents at Bazel, DigiCert, Microsoft, Let's Encrypt and Ericsson, the …
22 sources · 17 organisations · 4 postmortemsThe retry is the outage
A field guide to metastable failure: the class of outage where the system stays down after the thing that broke it has been repaired, because the ret…
30 sources · 19 organisations · 7 postmortems