Six published incidents between 2021 and 2025, at AWS, Meta, Slack, Cloudflare, Salesforce and Roblox, plus the source trails at Kubernetes, musl and Netflix, reconstruct how production systems actually turn service names into addresses and how that layer fails. After reading you can say, in a design review, which half of your naming system deserves the engineering attention, and why the obvious half is the wrong answer.
Every request your system makes begins by asking a shared, cached, eventually consistent database where to go. This page is about what happens when that database answers wrongly, and who writes the wrong answer.
State the problem without naming the technology and it sounds almost too small to study: a service has a name, a client needs an address, something in the middle translates. What makes the translation layer interesting is its shape. It is the only database almost every company runs that is written by one organisation, cached by thousands of others on terms the writer only partially controls, and consulted before any other dependency can even be reached. It sits underneath the monitoring, the deployment tooling and the VPN, which means it sits underneath the things you would use to fix it.
The public record on this layer is unusually good, because when it fails, it fails in front of everyone. This guide is built on six full incident write-ups: Amazon's October 2025 DynamoDB disruption, Meta's October 2021 disappearance, Slack's September 2021 DNSSEC outage, Cloudflare's July 2025 resolver outage, Salesforce's May 2021 multi-instance disruption, and Roblox's 73-hour Consul failure of October 2021, which is the same problem wearing a service-registry costume. Around them sit the build records: Stripe's published resolution path, the Kubernetes NodeLocal DNSCache design, Netflix's decision to bypass DNS entirely for internal traffic in 2012, and the musl libc project's ten-year argument about TCP fallback.
Here is the finding that reorganised this page. Going in, the expectation was a catalogue of overloaded resolvers and unreachable name servers, the read path buckling. That catalogue barely exists. In five of the six incidents the machinery that answers queries stayed healthy; what failed was the machinery that decides what the answers should be. Amazon's automation wrote an empty record. Meta's name servers withdrew themselves, on purpose, exactly as designed. Salesforce pushed a change that its own review process would have caught. Cloudflare attached production prefixes to a non-production topology and a later, unrelated change shipped that state worldwide. Only Slack's incident involved the serving side misbehaving, and even there the trigger was a rollout and the wound was kept open by caches faithfully doing their job. The read path is the most hardened infrastructure on the internet. The write path above it is ordinary software, and it is where this layer actually breaks.
The second finding follows from the first and is the one to carry into capacity planning: this layer heals at the speed of its caches, not at the speed of your fix. Slack's engineers rolled back within minutes; the 24-hour TTL on the DS record at the .com zone, and resolvers legitimately reusing a cached denial, decided how long users suffered. Caching is simultaneously the layer's great defence, since Moura et al. measured in 2018 that full resolver caches let about half of clients ride out an authoritative outage, and the reason a bad record outlives its correction. You cannot have one property without the other, and the decisions section below treats TTL as exactly that trade.
Scope. This guide covers internal and external service naming for systems that run on infrastructure they at least partly control: the resolution path, the record-writing control plane, TTL and negative-caching behaviour, and the DNS-versus-registry decision. It deliberately does not cover DNS security as a field (DNSSEC appears only through Slack's operational experience), DDoS defence of authoritative fleets, domain registration and registrar risk, or multicast/zero-conf discovery.
The resolution path that recurs across every published system, and the two components that vary: where the caches sit, and what writes the records.
Strip the branding from Stripe's, Kubernetes' and AWS's published paths and the same five-layer shape appears in all of them. A stub resolver inside the client process asks a cache on the same host; that cache forwards misses to a shared recursive fleet; the fleet asks the authoritative service; and, standing apart from all of it, a control plane computes what the authoritative service should be saying and writes it there through an API. Every layer exists for a measured reason, and each one is attributable.
The half of the system nobody budgets for. The stub lives inside your binary, chosen
by your base image, and its behaviour is load-bearing: Kubernetes ships
ndots:5, so a pod looking up an external name tries the cluster search
domains first, multiplying queries, as Marco Pracucci measured in 2019. Alpine images
inherited musl's UDP-only resolver until May 2023, so any answer over 512 bytes
simply failed.
Documented at: Pracucci, musl 1.2.4 notes
Stripe runs Unbound on every host; Kubernetes' KEP-1024 added a per-node cache after years of the 5-second-delay issue. The stated reasons are identical: keep queries off the network, and dodge the kernel's conntrack races by answering locally and using TCP upstream. This box exists because of a kernel bug class, not for hit rate.
Runs this way at: Stripe, Kubernetes
A cluster of resolvers holding the forwarding policy: which domains go to the cloud provider's resolver, which to internal zones, which to the public internet. Stripe's version keeps separate retry-timeout state per forwarding rule so one slow upstream cannot poison timeout calculations for the rest. This is where resolution policy is administered, and where it is observable.
Runs this way at: Stripe
Anycast fleets serving records, engineered to a different standard from everything else on this page: AWS sells Route 53's query answering with a 100% availability SLA, and AWS's Well-Architected guidance describes it as statically stable, able to keep answering with last-known state while its control plane is down. The evidence of the six incidents says this engineering largely works.
Documented at: AWS SLA, REL11-BP04
The planner-and-enactor pair from Amazon's postmortem is the general shape: something watches health and intent, computes the desired record set, and applies it through an API. It is deliberately redundant, which is exactly what made it dangerous: two enactors with no ordering between them is a distributed system, whether or not it was designed as one.
Documented at: AWS, 2025, Salesforce, 2021
Netflix's Eureka and Roblox's Consul replace the DNS path for internal calls with a registry: instances heartbeat in (every 30 seconds at Netflix, eviction after about 90), clients cache the whole registry and keep working from that cache if the registry dies. Fresher health data, bought by operating one more tier-0 stateful system. Section 3 takes the choice apart.
Two reconstructions worth flagging as mine rather than reported. First, no published account fully describes the write path's concurrency control; Amazon's postmortem is the closest, and what it describes is enactors checking plan freshness at start and then applying without a fence, which is a read-check-write race textbook. Second, the per-host cache layer is described by Stripe and Kubernetes but only implied elsewhere; treat "every host caches" as corroborated for container platforms and inferred for everything else.
Four forks in the road, each with the recorded argument and the condition that flips the answer. The fourth one, almost nobody makes deliberately.
serve_stale.| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Internal discovery | Registry (Netflix), DNS (Kubernetes) | The other one | Eviction speed vs. one more tier-0 system | Eureka docs, KEP-1024 |
| TTL policy | Short + health checks at leaves | Long everywhere | Failover speed; hit-rate cost is small (2001) | Jung et al., Moura 2019 |
| Expiry under outage | Serve stale, 7-day cap | Strict expiry | Akamai ran it since 2011; availability won | RFC 8767, CoreDNS #4309 |
| Where to fix clients | Node-level cache + TCP | Wait for libc | Could not ship a glibc/musl/kernel fix | k8s #56903, musl thread |
| Failover mechanism | Health checks (data plane) | Emergency record edits | Control plane is exactly what fails under stress | AWS REL11-BP04 |
One decision the record shows almost nobody making deliberately: how the record-writing automation serialises its writers. Amazon's postmortem describes redundant enactors racing; Salesforce's describes an emergency change lane that skipped review; Cloudflare's describes a config store where a dormant error waited for an unrelated change to ship it globally. Three different companies, one missing decision: what orders and gates writes to the layer everything trusts. If your zone automation has two writers and no fencing, you have made Amazon's choice without Amazon's postmortem.
Six incidents, three failure classes. The class names are mine; the incidents keep sorting themselves into them.
Reading the six write-ups together, each incident lands in one of three classes. Class one: the writer corrupts the record. The serving fleet is healthy and faithfully serves a wrong answer that automation or a change process wrote (AWS, Salesforce, Cloudflare). Class two: the system depends on itself. The failure disables the tools needed to repair it, because naming sits under the tooling (Meta, Roblox). Class three: the cache preserves the wound. The fix ships fast and the installed base of caches keeps serving the failure (Slack, and the long tail of the AWS recovery). Class one is where prevention pays; class two is where recovery planning pays; class three is where you mostly pay in advance, by choosing TTLs.
The class-two incidents deserve one more sentence, because they rhyme across a decade of this material: Meta could not reach its tools because the tools resolved through the thing that failed, and Roblox could not see Consul because the telemetry resolved through Consul. No public postmortem in this corpus describes the inverse, a naming failure contained because the repair path was verified independent. Either nobody builds that verification, or nobody who builds it has needed to write a postmortem. Both readings argue for a game day that severs naming and then tries to use your own consoles.
Scale, latency and cost figures from the sources, each dated. The planning paragraph below separates what was measured from what was claimed.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Public resolver query volume | ~1.9T/day | Cloudflare 1.1.1.1 | Scale of the thing that vanished for 62 minutes | early 2025 | ISOC Pulse |
| Resolver outage duration | 62 min | Cloudflare | Global withdrawal from one config refresh | 2025-07 | postmortem |
| Record-to-repair time | ~3 h | AWS DynamoDB | Empty record 11:48 PM to manual repair ~2:40 AM | 2025-10 | postmortem |
| Dependent-service recovery tail | +12 h | AWS us-east-1 | Services atop DynamoDB recovered through 2:20 PM | 2025-10 | ThousandEyes |
| Total outage, self-dependent recovery | ~6 h | Meta | Physical access required to repair remotely-run network | 2021-10 | postmortem |
| Discovery-layer outage | 73 h / ~$25M | Roblox | Consul contention + BoltDB pathology; company's own figures | 2021-10 | postmortem, DCF |
| Rollback horizon, parent zone | 24 h | Slack / .com | DS-record TTL that bounded DNSSEC un-rollout | 2021-09 | postmortem |
| Latency lever of TTL | 183 → 28.7 ms | .uy ccTLD | Median latency after raising TTL from 5 min to 1 day | 2019 | Moura, IMC '19 |
| Cache protection during outage | ~50% of clients | measured testbed | Ride out an authoritative outage with full caches; retries amplify traffic up to 8× | 2018 | Moura, IMC '18 |
| Lookups receiving no answer | 23% | MIT trace | Failure traffic dominated DNS packets even in 2001 | 2001 | Jung et al. |
| Client-side stall, conntrack race | 5 s (up to 30 s) | Kubernetes | Dropped UDP packet waits out resolver timeout; 3 retries × (A + AAAA) | 2019 | KEP-1024 |
| Registry eviction clock | 30 s / ~90 s | Netflix Eureka | Heartbeat interval / eviction after missed renewals | checked 2026 | Eureka docs |
| Serve-stale ceiling | 7 days | IETF | Suggested cap on serving expired records during outage | 2020 | RFC 8767 |
| Data-plane availability SLA | 100% | AWS Route 53 | Query answering only; record changes carry no such SLA | checked 2026 | SLA |
Measured, from primary sources: the outage durations, the Moura and Jung figures, the KEP timings. Claimed, by the operator: Roblox's $25M, the 1.9T queries/day, and every SLA (an SLA is a refund schedule, not a measurement). Derived, by me: the "+12 h" AWS tail is the gap between the postmortem's repair time and its full-recovery time, corroborated by external monitoring. The 2001 MIT failure rate is a different internet; keep it as an order-of-magnitude reminder that failure traffic dominates, not as a current figure. No public source in this corpus prices the record control plane itself; what a planner/enactor pipeline costs to run safely is an open number.
Every source behind this page, graded. Filter by kind.
The most detailed public description of a DNS record control plane anywhere: planner, redundant enactors, the race, the empty record, and the admission that the automation was disabled worldwide afterwards.
The self-dependency incident: DNS servers designed to withdraw their BGP routes on lost backbone connectivity all did so at once, and the repair tooling resolved through the thing that failed.
A vendor bug in NSEC synthesis for wildcards let resolvers cache a denial of slack.com's existence; the rollback was fast and irrelevant, because caches held the wound open for up to 24 hours.
A config error dormant since June was shipped globally by an unrelated change's refresh; 62 minutes without the world's second-largest public resolver.
A DNS change pushed through the emergency lane without an emergency; servers failed to restart after a zone transfer. Unusually frank about the process failure.
The registry version of this failure: one Consul cluster under everything, a streaming feature and a BoltDB pathology, and telemetry that depended on the system it watched. 73 hours.
The accepted design that ended the 5-second-delay era: per-node cache, conntrack bypass, TCP upstream. The motivation section is a compressed history of everything wrong with in-cluster DNS.
Standardises answering with expired records when the authority is unreachable, suggested cap seven days, and records Akamai running it in production since 2011.
DNS as the default name system, deliberately behind a plugin interface so deployments can substitute xDS or registry resolvers per channel.
The thread where a platform's worth of users converged on the same symptom and traced it to kernel conntrack races on parallel UDP queries, not to any DNS server.
Serve-stale demand arriving in cluster DNS by RFC number; the cache plugin gained
serve_stale as a result.
The recorded argument for the rejected approach: truncated UDP answers treated as acceptable, bounded, and cheaper than TCP round trips. Every Alpine image carried the consequence.
The reversal, in the project's own words: "fixing the longstanding inability to query large DNS records". A decade of ecosystem pain closed in one release.
Netflix's own statement of why internal traffic bypasses DNS: registry with 30-second heartbeats, ~90-second eviction, and clients that survive registry outages on cached state.
The foundational measurement: cutting A-record TTLs to a few hundred seconds barely hurts hit rates, NS-record caching is what protects the system, and 23% of traced lookups got no answer at all.
TTL as a latency lever, measured: one ccTLD's raise from five minutes to a day cut median latency from 183 ms to 28.7 ms, and the operator changed production after seeing the data.
Caching quantified as the DNS's principal outage defence: full caches carry about half of clients through an authoritative outage, while client retries multiply load on the struggling authority up to 8×.
The conference version of the postmortem, with the attempt-by-attempt history the blog compresses; the title alone records that two aborted rollouts preceded the incident. Slides linked from the session page.
The IMC '18 findings presented to the operator community that runs the resolvers in question; the slide deck is the fastest way into the caching-as-defence data.
The clearest published internal resolution path: Unbound on every host and as a shared fleet, per-domain forwarding rules, per-rule timeout state, and the metrics pipeline watching it all.
The kernel-level diagnosis behind the 5-second delays, written by the engineer who then fixed part of it upstream: parallel A and AAAA queries over one socket race in conntrack insertion and one packet is silently dropped.
The measurement that made ndots:5 famous: every external lookup from a
default pod walks the search domains first, multiplying queries and latency.
Independent expert corroboration of the Slack mechanism: the NSEC type-bitmap error and why aggressively-caching validating resolvers kept denying slack.com after the rollback.
The outside view of Meta's incident: global BGP monitoring watched the nameserver prefixes withdraw minutes before resolution failed worldwide.
Puts the 62 minutes in ecosystem terms: ~1.9 trillion queries a day, and the plain warning that one upstream resolver, however good, is a single point of failure.
The 100% availability SLA on query answering, and, by its silence, the absence of any equivalent commitment on record changes. The asymmetry is the thesis of this page in contract form.
AWS's own doctrine: Route 53's control plane lives in us-east-1, its data plane in 200+ PoPs, and failover should ride health checks, never emergency record edits. Published months before the October 2025 incident tested the write path.
Independent external timeline of the DynamoDB incident, corroborating the repair time and the long dependent-service tail after the record was fixed.
Six rungs from an evening's toy to a production-shaped drill. The line from reading to skill is crossed at rung four, where you reproduce Amazon's race on your laptop.
Stand up Unbound or CoreDNS as a recursive resolver, point dig through
it, and watch the same query answered by the authority, then by the cache. Lower a
record's TTL at a zone you control and watch the change propagate.
Done when: you can show, with timestamps, a cached answer surviving after the authoritative record changed. Teaches: the cache hierarchy is the system; the authority is just its source of truth.
Firewall your resolver off from the authority mid-run. Watch answers keep flowing
until TTL expiry, then fail. Enable serve-expired (Unbound) or
serve_stale (CoreDNS) and repeat.
Done when: the same outage produces SERVFAIL in one config and stale answers in the other, and you can say which your production resolvers do today. Teaches: RFC 8767's trade, felt rather than read.
In a default Kubernetes pod, tcpdump port 53 while resolving an external
name. Count the search-domain expansions from ndots:5 and the parallel
A/AAAA pairs. Then add a trailing dot and count again.
Done when: you have the two packet counts side by side. Teaches: half the resolution system ships inside your base image, configured by someone else.
Write a planner emitting numbered zone plans and two enactors applying them to a local zone file or test zone, with a cleanup step that deletes old plans. Add a random stall to one enactor. When the empty-record state appears, fix it properly: monotonic plan numbers checked at apply time, and cleanup that refuses to delete the active plan.
Done when: the stale writer's apply is rejected by version check, under a stress loop, every time. Teaches: the write path is a distributed system, and Amazon's October 2025 failure mode fits in 200 lines.
Run a toy registry (heartbeat in, JSON out, client cache) next to a DNS zone with 60-second TTLs, both fronting the same two backends. Kill a backend and measure time-to-eviction on each path; then kill the registry and watch clients coast on cached state.
Done when: you have both eviction curves on one chart and can defend a choice between them for your own workload. Teaches: the freshness-versus-operability trade that separates Netflix's choice from Kubernetes'.
In staging, make internal DNS lie: empty answers for one service, NXDOMAIN for another, five-second delays for a third. Watch which clients retry, which cache the failure, which crash. Then the class-two drill: with resolution down, try to reach your own dashboards, runbooks and cloud consoles.
Done when: you have a written inventory of what breaks, and at least one repair tool verified to work with naming down. Teaches: Meta's and Roblox's lesson at staging prices; failure here is not binary, and the repair path is part of the architecture.
The queries that found the material above, so the reader can extend the search when this page goes stale.
"DNS Enactor" "DNS Planner" race conditionsite:engineering.fb.com outage BGP DNS "disable those BGP advertisements""what happened during" DNSSEC rollout NXDOMAIN negative caching"multi-instance service disruption" DNS "emergency break fix""return to service" Consul BoltDB contention postmortemkubernetes "DNS intermittent delays of 5s" conntrack insert_failed"racy conntrack and dns lookup timeouts""ndots:5" resolv.conf performance kubernetesmusl "tcp fallback" DNS truncated 512 bytes"serve stale" RFC 8767 resolver production Akamai"cache me if you can" DNS TTL IMC"when the dike breaks" DNS caching DDoS"effectiveness of caching" DNS trace TTLeureka wiki "may not be healthy or may not even exist"grpc naming.md resolver plugin dns"NodeLocal" dnscache KEP conntrack NOTRACK motivation"data plane" "control plane" route 53 static stability failoverThe pattern behind the queries: exact phrases learned from one source ("DNS Enactor", "insert_failed", "serve stale") retrieve the layer of material the generic query never surfaces. Search the vocabulary a system taught you, not the name of the technology.
Access note: this guide was researched from an environment whose sandbox could fetch GitHub-hosted sources directly; all other pages were located, date-confirmed and quote-verified through search-engine retrieval of the cited URL on the date shown. Quotes are reproduced only where the cited page's own text was returned verbatim.