In the 2017 AWS S3 outage, the status page could not report the outage because it depended on S3. What does that tell you about designing status and control systems?

The case, as publicly reported

On 28 February 2017, an authorised engineer running an established playbook to debug an S3 billing issue in us-east-1 entered a command with a typo. It removed a substantially larger set of servers than intended, including capacity supporting the index and placement subsystems. Both required a full restart, which had not been done at that scale in years and took hours.

The visible detail that became the lesson: the AWS Service Health Dashboard itself depended on S3 in that region, so for the first hours AWS could not update its own status page and had to post to Twitter instead.

What it tells you

1. A status system must not share a failure domain with the thing it reports on. This sounds obvious and is violated constantly — status pages hosted on the same cloud, alerting that emails through the affected mail system, incident chat that runs on the affected platform, runbooks in a wiki behind the failed SSO.

The rule: out-of-band by design. Different provider, different region, different dependency chain. It is cheap, because a status page is a static file.

2. Control planes and data planes must fail independently. The recovery took hours because restarting the subsystems required the very infrastructure that was down. Static stability — the property that a system keeps operating on its existing state when its control plane is unavailable — is the principle AWS itself now writes about extensively, and it came from incidents like this.

3. Blast radius is a design parameter, not an outcome. The command was legitimate; the tooling allowed it to affect more than intended. Modern practice is to limit what any single operation can remove: require confirmation above a threshold, cap the percentage of capacity a command can touch, partition into cells so that no single operation spans all of them.

4. Restart paths decay if never exercised. The subsystems had grown while their full restart had not been tested at scale. Anything whose recovery path is untested has an unknown RTO.

What I would design differently

• Status page on a different provider, generated statically, with a manual publish path.
• Incident communications with an out-of-band fallback (a channel not dependent on the estate).
• Cell-based partitioning so an operator error is bounded to one cell.
• Scheduled recovery drills that exercise full restart, not just failover.

What a strong answer adds

Recognising the recursive version of the question: what does our incident response depend on? Most teams have never listed it, and the answer usually includes SSO, a chat platform, a cloud console, and a wiki — several of which may be inside the blast radius of the incident they are responding to.