Slack 2021: When Autoscaling Cannot Keep Up
also called Slack January 2021 Outage
The first Monday back after the holidays produced a traffic ramp that outpaced the scaling behaviour of a managed network component, and the degradation cascaded.
What happened
On 4 January 2021 — the first working Monday after the holiday period — Slack suffered a widespread outage. Their published postmortem is unusually detailed and worth reading in full.
The shape of it: traffic ramped steeply as a global user base returned to work simultaneously, after a period of unusually low weekend and holiday traffic. Their AWS Transit Gateways, which scale automatically, could not scale quickly enough to match the ramp. Network saturation caused packet loss between availability zones.
The packet loss then interacted badly with the rest of the system. Health checks began failing for instances that were actually healthy, service discovery churned, and load was redistributed onto components that were themselves degraded. A network capacity problem became a systemic one.
The design lessons
Autoscaling has a reaction time, and that reaction time is a design assumption. A system whose capacity plan depends on scaling must have enough standing capacity to survive the ramp until scaling catches up — and the worst ramps follow periods of low traffic, precisely when everything has scaled down.
Managed components scale according to their own rules, not yours. The scaling behaviour of a managed network component, a managed database or a serverless concurrency limit is part of your capacity model, and its ramp rate is frequently undocumented until you exceed it.
Health checks that fail under network degradation amplify the problem. An instance marked unhealthy because a probe was dropped is removed from service, concentrating load on the remainder, which makes the congestion worse. Health checking needs to distinguish a failed instance from a failed network path.
The transferable lesson
Model the worst ramp, not the peak. Traffic that goes from a holiday trough to a Monday peak is a harder engineering problem than sustained high traffic, because it is exactly when the system has the least standing capacity.
Pre-scaling ahead of predictable events — the first Monday of a year, a marketing launch, a seasonal peak — is unglamorous and repeatedly proves cheaper than relying on reaction.