AWS S3 2017: The Blast Radius of a Typo
also called S3 us-east-1 Outage
A mistyped command during routine debugging removed far more capacity than intended, and the affected subsystems had not been restarted in years.
What happened
On 28 February 2017 an engineer following an established playbook to remove a small number of servers from an S3 billing subsystem in us-east-1 entered the command incorrectly. A larger set of servers was removed than intended, including servers supporting the index subsystem — which holds metadata and location information for all S3 objects in the region — and the placement subsystem.
Both required a full restart. Neither had been fully restarted in years, so the restart and its safety checks took far longer than anyone expected. S3 was substantially unavailable in us-east-1 for several hours, and because so much of the internet depends on that region, the effects were extremely visible.
There was a second lesson in the aftermath: the AWS status dashboard itself depended on S3 in us-east-1, so AWS could not update customers through their normal channel during the outage.
The design failures
Tooling with unbounded blast radius. A command capable of removing an arbitrarily large fraction of a critical subsystem's capacity, with no confirmation step and no limit on what a single invocation could do.
Untested restart paths. The subsystems worked continuously for years, so their cold-start behaviour was effectively unknown. Recovery time was not a measured property.
Circular dependency in the status channel, so the communication mechanism failed with the thing it was meant to report on.
The changes that followed
AWS added guardrails so removal tooling cannot take capacity below a minimum threshold, and it removes capacity more slowly. They partitioned the index subsystem into cells to reduce restart time and blast radius, and moved the status dashboard off its dependency.
The transferable lesson
Three questions worth asking of your own estate. What is the maximum damage a single operator command can do, and is there a limit enforced by the tool rather than by the operator's care? When did this system last cold start, and how long did it take? And does your incident communication path depend on the systems it reports on — status page, alerting, chat, runbook storage, authentication?
The last one catches a surprising number of organisations, and it costs almost nothing to fix in advance.