A configuration change disconnects a company's internal network. Engineers cannot access the tooling needed to revert it. Analyse.
Show the full answer Hide the answer
What the interviewer is testing
Whether you map the dependencies of the recovery path, not just of the service.
The failure shape
A change removed connectivity. The tools needed to diagnose and revert — remote access, internal tooling, monitoring, possibly identity and even physical access systems — all depended on the network that had just disappeared.
So a configuration error that would take minutes to reverse took hours, because reaching the equipment required physical presence and manual work.
Meta's 2021 outage followed exactly this shape, with an additional twist worth noting: their DNS servers were designed to withdraw their route advertisements when they lost connectivity to the data centres — a sensible behaviour that, combined with a total backbone loss, removed their domains from the internet's routing tables entirely.
The findings
No out-of-band access path. Recovery required the production network.
Circular dependency in every supporting system. Identity, tooling, runbooks, monitoring and communications all inside the failure domain.
An automated safeguard that did not stop the command, so the last line of defence had a defect nobody had tested.
Designed behaviours interacting badly. Each component behaved correctly in isolation; the combination was catastrophic — which is why systemic reasoning matters more than component correctness.
The changes
An out-of-band management path sharing nothing with the production estate — separate connectivity, separate credentials, separate identity. It sounds paranoid until the day it is the only thing that works, and it is almost never tested.
Runbooks and credentials available offline, not only in a wiki inside the failure domain.
Test the safeguards, since a guard that has never been exercised is an assumption.
Rehearse the total-loss scenario, including how the team communicates when internal chat is down.
What a strong answer adds
The checklist to run in any organisation: does your VPN or bastion depend on the production network? Your identity provider? Your alerting and paging? Your runbook storage? Your chat? Your status page? Your configuration management? Your DNS?
Most organisations discover several yes answers, and each is cheap to fix in advance.
Common weak answers
More careful change management for network changes. Better documentation, stored in the system that was unavailable.