Slack's Cellular Migration
After repeated availability-zone-level incidents, Slack rebuilt its infrastructure into per-zone cells with the ability to drain traffic away from a failing zone in minutes.
The problem they identified
Cloud availability zones do not only fail cleanly. Slack's published account describes the harder case: a zone that is partially degraded — elevated packet loss, slow disks, a struggling dependency — rather than plainly down.
Their pre-existing architecture had services spread across zones with cross-zone calls throughout. In that arrangement a degraded zone poisons everything: requests entering healthy zones still make calls into the sick one, so the failure is not contained by the zone boundary at all. Worse, cloud health signals often do not flag a grey failure, so automated systems keep routing into it.
The change
Cellular architecture aligned to availability zones. Each cell is a full service stack within one zone, and requests are served within the cell rather than fanning across zones. Cross-zone traffic on the request path is treated as something to eliminate, not merely to tolerate.
The critical operational capability is traffic drainage: the ability to shift all traffic away from one cell quickly and deliberately. Rather than diagnosing what exactly is wrong in a degraded zone — which takes hours — the response becomes "drain it, then investigate", which takes minutes.
The lessons
1. Grey failure is the hard case. Systems are usually designed for a component being up or down. Partial degradation defeats health checks, defeats automated failover, and is the most common real failure mode at scale.
2. Mitigation should not require diagnosis. The most valuable operational capability is a generic, well-rehearsed action that resolves a broad class of problems without knowing which one you have. Draining a cell is one; rolling back a deploy is another. Both trade precision for speed, and during an incident that is the right trade.
3. Cross-zone calls are a liability as well as a cost. They are usually discussed as a data-transfer expense. The availability argument is stronger: every cross-zone call spreads one zone's problems into another zone.
4. Isolation must be enforced end to end. A cell that is nominally isolated but calls a cross-zone shared dependency is not isolated. The hard part of this migration is finding all such dependencies.