Meta 2021: The Outage That Locked Out Its Own Engineers
also called Facebook October 2021 Outage
A backbone configuration command withdrew Facebook's BGP routes globally, and the tools needed to fix it depended on the network that had just disappeared.
What happened
On 4 October 2021, a command issued during routine maintenance to assess backbone capacity unintentionally disconnected all of Facebook's backbone connections between its data centres. An audit tool that should have prevented the command had a bug and did not stop it.
The cascade that followed is the interesting part. Facebook's DNS servers are designed to withdraw their BGP route advertisements if they lose connectivity to the data centres — a sensible design that stops a DNS server serving answers it cannot back up. With the backbone gone, every DNS server did exactly that, so the authoritative name servers for facebook.com disappeared from the internet's routing tables entirely.
At that point Facebook, Instagram and WhatsApp were not merely down; their domains did not resolve anywhere in the world. Recovery took roughly six hours.
The circular dependencies
Remote access to the affected equipment ran over the network that was down. Internal tooling, including the systems engineers would use to diagnose and revert, was inaccessible. Reports indicated that even physical access was complicated because badge and access systems were affected.
Engineers had to reach data centres physically and work on hardware directly, which is why a configuration error took hours rather than minutes to reverse.
The transferable lesson
Map the dependencies of your recovery path, not just of your service. The question is not only "what breaks when X fails" but "what do we use to fix X, and does it depend on X".
The list worth checking in any organisation: does your VPN, bastion or remote access depend on the production network? Does your identity provider? Your alerting and paging? Your runbook storage? Your chat? Your status page? Your configuration management? Your DNS?
An out-of-band recovery path — one that shares nothing with the production estate — sounds paranoid until the day it is the only thing that works. It is cheap to establish and it is almost never tested.