Platform & Infrastructure 04 Sep 2026 32 min read

When name resolution fails, the record was wrong, not the server

How production systems turn service names into addresses, why the control plane that writes the records fails more often than the servers that answer queries, and why recovery runs at the speed of the caches.

Six published incidents (AWS October 2025, Meta 2021, Slack 2021, Cloudflare 2025, Salesforce 2021, Roblox 2021) plus the build records at Stripe, Kubernetes, musl and Netflix, reconstructed into a reference resolution path, four decision tables, a three-class failure catalogue and a build ladder. After reading, an architect can say which half of their naming layer deserves the engineering attention, compute a rollback horizon before shipping a DNS change, and defend a DNS-versus-registry choice with evidence.

The finding that surprised me

In five of six major naming outages the query-serving infrastructure stayed healthy; what failed was the automation and change process that decides what the answers should be, and the caches then made the wrong answers durable.

What you get out of it

  • Name resolution fails from its write path: automation races (AWS), deliberate self-withdrawal (Meta), bypassed change review (Salesforce) and dormant config shipped by an unrelated refresh (Cloudflare) broke it; the anycast serving fleets largely did not.
  • Recovery runs at the speed of caches you do not control: Slack rolled back in minutes and waited on a 24-hour DS-record TTL; compute the rollback horizon before shipping any naming change.
  • A redundant record-writer is a distributed system: two enactors without fencing or monotonic versions is Amazon's October 2025 failure mode, reproducible in 200 lines.
  • The repair path must not resolve through the thing it repairs: Meta's tooling and Roblox's telemetry both depended on the failed naming layer, converting outages into marathons.
  • Caching is both the defence and the wound: full caches carry about half of clients through an authoritative outage (IMC 2018), and the same memory preserves bad records after the fix.

Scope

Why this, now. The October 2025 DynamoDB outage published the most detailed account of a DNS record control plane ever released, and it reframes a decade of naming incidents as write-path failures.

What it does not cover. DNS security as a field (DNSSEC appears only through Slack's operational experience), DDoS defence of authoritative fleets, registrar and domain-registration risk, and multicast/zero-conf discovery.

Open the field guide → Self-contained: it loads nothing at read time, follows your system theme, and prints cleanly.