In the October 2021 Meta outage, engineers reportedly struggled to physically access equipment because the access-control systems were also down. What class of dependency is that, and how do you find yours?

The case, as publicly reported

On 4 October 2021, a command issued during routine backbone capacity assessment took down all connections between Meta's data centres. Their DNS servers are designed to withdraw BGP route advertisements when they lose backbone connectivity — a reasonable health signal — so they withdrew, and the authoritative name servers for facebook.com became globally unreachable.

The compounding factor: internal tooling used to diagnose and repair was itself reachable only through the same infrastructure, and building access and conference systems were affected. The outage lasted roughly six hours.

The class of dependency

Circular operational dependency: the mechanism used to recover from a failure depends on the thing that has failed. It is a distinct failure class from ordinary dependency, because it does not degrade service — it removes your ability to restore service, which converts a short outage into a long one.

Common instances, nearly all of which exist somewhere in most estates:

• The runbook is in a wiki behind the SSO that is down.
• The deployment pipeline needs the cloud API that is failing.
• Alerting emails through the mail platform that is affected.
• The VPN needed to reach the console authenticates against the directory that is down.
• Secrets needed to rebuild are stored in the system being rebuilt.
• The incident bridge runs on the platform that is out.
• Physical access to the data centre requires a badge system on the corporate network.

How to find yours

Run the exercise explicitly. Take each major system and ask: if this is completely unavailable, what can we no longer do? — then specifically, can we still diagnose, communicate, authenticate, deploy and enter the building?

The stronger version is a game day that removes a foundational service in a controlled way and observes what the response team cannot do. Teams discover their circular dependencies in minutes, where a document review misses most of them.

What good looks like

• Break-glass credentials held offline, in a safe, with an audited checkout process.
• Out-of-band management: a separate network path to console access that does not traverse the production network.
• Runbooks exported to a location reachable with nothing but a laptop and a phone.
• An incident channel on a different provider, agreed in advance so nobody has to decide during the incident.
• DNS resilience: a second provider, and careful thought about health signals that withdraw routes — the automation was working as designed, which is the uncomfortable part.

What a strong answer adds

Noting that the failure was not a bug. The DNS servers withdrawing routes on loss of connectivity is correct behaviour in isolation; it became catastrophic in combination with a total backbone loss. That is a systems-thinking failure — an interaction between two individually-correct mechanisms — and it is the kind that code review never catches.