Platform & Infrastructure 29 Aug 2026 42 min read

Keeping one failure from reaching everyone

How production systems partition themselves so one failure reaches a fraction of users, and why the containment keeps leaking through whatever stayed global.

Slack, Shopify, Salesforce, DoorDash, Amazon EBS and AWS all cut their systems into independent replicas so that one fault hurts some customers rather than all of them, and all of them publish how. This guide reconstructs the common architecture across those six, then shows from five public postmortems why the pattern keeps failing anyway: the partitioning covers the request path, while the faults arrive through the change path, a plane that is almost always still global. A reader finishes with a decision table for cell boundary, size and routing, a six-rung build ladder that crosses from cells to change-plane waves, and the specific questions to ask of their own pipeline on Monday.

The finding that surprised me

The 2020 Physalia paper had already identified that cells do not contain deployments and had built a second, orthogonal partition for it, called colours, that operates on the change path rather than the request path; every large outage since is a failure of exactly that second partition, and none of the four organisations involved had one.

What you get out of it

  • Cells partition the request plane; the change plane, which carries config, content, DNS, packages and code, is a second axis that almost nobody partitions, and it is where every large recent outage propagated.
  • No published postmortem in this corpus describes a single cell failing and the failure staying inside it, which means the public record is structurally incapable of telling you whether your containment works; only a production drain can.
  • The drain, not the cell, is the thing an incident commander uses. Slack targeted five minutes and achieved 60 seconds with 1% weight granularity; without a fast drain, cells buy a smaller incident rather than a shorter one.
  • Kubernetes zone affinity is a cost preference, not an isolation boundary. It silently withholds hints when any node lacks a zone label or allocatable CPU, and its own successor KEP calls the heuristic unpredictable.
  • AWS's published reference implementation deploys to a sandbox cell, checks aliveness, then updates all other cells, which is the exact change-plane anti-pattern its own Well-Architected guidance warns against.

Scope

Why this, now. Four of the five largest published outages of the last three years, Cloudflare and AWS in late 2025, CrowdStrike in 2024 and Datadog in 2023, propagated through unpartitioned change machinery rather than through anything cellular, making this the failure class an architect is most likely to meet next.

What it does not cover. Multi-region active-active failover, sharding purely for scale, single-tenant per-customer deployments, the security case for tenant isolation, and Kubernetes multi-cluster management tooling.

Open the field guide → Self-contained: it loads nothing at read time, follows your system theme, and prints cleanly.