When name resolution fails, the record was wrong, not the server
How production systems turn service names into addresses, why the control plane that writes the records fails more often than the servers that answer queries, and why recovery runs at the speed of the caches.
Six published incidents (AWS October 2025, Meta 2021, Slack 2021, Cloudflare 2025, Salesforce 2021, Roblox 2021) plus the build records at Stripe, Kubernetes, musl and Netflix, reconstructed into a reference resolution path, four decision tables, a three-class failure catalogue and a build ladder. After reading, an architect can say which half of their naming layer deserves the engineering attention, compute a rollback horizon before shipping a DNS change, and defend a DNS-versus-registry choice with evidence.
In five of six major naming outages the query-serving infrastructure stayed healthy; what failed was the automation and change process that decides what the answers should be, and the caches then made the wrong answers durable.
What you get out of it
- Name resolution fails from its write path: automation races (AWS), deliberate self-withdrawal (Meta), bypassed change review (Salesforce) and dormant config shipped by an unrelated refresh (Cloudflare) broke it; the anycast serving fleets largely did not.
- Recovery runs at the speed of caches you do not control: Slack rolled back in minutes and waited on a 24-hour DS-record TTL; compute the rollback horizon before shipping any naming change.
- A redundant record-writer is a distributed system: two enactors without fencing or monotonic versions is Amazon's October 2025 failure mode, reproducible in 200 lines.
- The repair path must not resolve through the thing it repairs: Meta's tooling and Roblox's telemetry both depended on the failed naming layer, converting outages into marathons.
- Caching is both the defence and the wound: full caches carry about half of clients through an authoritative outage (IMC 2018), and the same memory preserves bad records after the fix.
Scope
Why this, now. The October 2025 DynamoDB outage published the most detailed account of a DNS record control plane ever released, and it reframes a decade of naming incidents as write-path failures.
What it does not cover. DNS security as a field (DNSSEC appears only through Slack's operational experience), DDoS defence of authoritative fleets, registrar and domain-registration risk, and multicast/zero-conf discovery.
Other field guides
Everything that survived has someone else's name on it: ten years of Lyft's platform
Reads eleven of Lyft's published platform components through git history, governance files and the package, container and module registries that date…
22 sources · 7 organisations · 2 postmortemsWhen one platform runs everything: eleven years of Monzo's architecture
A single-company dig, reconstructed from Monzo's own incident reports and engineering posts, its conference talks, and the archive banners and commit…
31 sources · 19 organisations · 3 postmortemsChanging what the services stand on: ten years of ByteDance, read from its own repositories
An organisation past a certain size cannot ask its services to change, so when the layer beneath them becomes the constraint it replaces that layer a…
24 sources · 7 organisations · 5 postmortems