Evidence ledger
28 sources
Checked 04 Sep 2026
Evidence ledger
One row per claim in When name resolution fails, the record was wrong, not the server: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how production systems turn service names into addresses, why the control plane that writes the records fails more than the servers that answer queries, and why recovery runs at the speed of the cache.
Checked date for all rows: 2026-09-04. Access notes: this session's sandbox allows direct fetches only to GitHub-hosted content and package registries; all other sources were located, date-confirmed and quote-verified through search-engine retrieval of the cited page (marked "search-verified" below). No quote appears here that was not returned verbatim from the cited page or from the page's own published excerpts.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | AWS | Summary of the Amazon DynamoDB Service Disruption in US-EAST-1 | postmortem | 2025-10 | 2026-09-04 | https://aws.amazon.com/message/101925/ | DynamoDB's DNS is maintained by a two-part automation: a DNS Planner and redundant DNS Enactors applying plans via Route 53 | "a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record for the service's regional endpoint" (search-verified) |
| 2 | AWS | Same | postmortem | 2025-10 | 2026-09-04 | https://aws.amazon.com/message/101925/ | The race: a delayed Enactor applied a stale plan over a newer one; the cleanup then deleted the plan in use, emptying the record and wedging the automation | One Enactor experienced "unusually high delays"; the clean-up "deleted the older plan as stale, immediately removing all IP addresses for the regional endpoint" and left "an inconsistent state that prevented further automated updates" (search-verified) |
| 3 | AWS | Same | postmortem | 2025-10 | 2026-09-04 | https://aws.amazon.com/message/101925/ | Repair was manual, and AWS switched the automation off globally afterwards | AWS "has disabled the DynamoDB DNS Planner and DNS Enactor automation worldwide" pending safeguards; incident began 11:48 PM PDT Oct 19, DNS record repaired ~2:40 AM, dependent-service recovery ran to 2:20 PM PDT (search-verified) |
| 4 | Meta | More details about the October 4 outage | postmortem | 2021-10-05 | 2026-09-04 | https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/ | Meta's DNS servers deliberately withdraw their BGP routes when they cannot reach the data centers; a backbone-wide disconnection made every DNS server do this at once | "our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection" (search-verified) |
| 5 | Meta | Same | postmortem | 2021-10-05 | 2026-09-04 | https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/ | The trigger was an audit command whose safety check failed | a command "with the intention to assess the availability of global backbone capacity ... unintentionally took down all the connections in our backbone network"; "a bug in that audit tool prevented it from properly stopping the command" (search-verified) |
| 6 | Meta | Same | postmortem | 2021-10-05 | 2026-09-04 | https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/ | Recovery required physical access because remote tooling depended on the failed name/network layer; outage ran roughly six hours | data centers are "hard to get into, and once you're inside, the hardware and routers are designed to be difficult to modify even when you have physical access" (search-verified) |
| 7 | Slack | What happened during Slack's DNSSEC rollout | postmortem | 2021-11 | 2026-09-04 | https://slack.engineering/what-happened-during-slacks-dnssec-rollout/ | The outage mechanism was a Route 53 NSEC bug for wildcard records: resolvers cached a record that falsely denied slack.com record types, so rollback did not immediately heal | error in "the NSEC type bitmap produced by the Route 53 server for wildcard domains", causing "some resolvers to reuse this cached NSEC record" and answer NXDOMAIN (search-verified; corroborated by [21]) |
| 8 | Slack | Same | postmortem | 2021-11 | 2026-09-04 | https://slack.engineering/what-happened-during-slacks-dnssec-rollout/ | Full recovery waited on caches Slack did not control, including the 24-hour DS record TTL at .com | "As public resolver caches were flushed and as the 24-hour TTL on the DS record at the '.com' zone expired, the error rates went back to normal"; third rollout attempt, Sept 30 2021 (search-verified) |
| 9 | Cloudflare | Cloudflare 1.1.1.1 incident on July 14, 2025 | postmortem | 2025-07 | 2026-09-04 | https://blog.cloudflare.com/cloudflare-1-1-1-1-incident-on-july-14-2025/ | A dormant config error from June linked 1.1.1.1 prefixes to a non-production topology; an unrelated change triggered a global refresh that withdrew the prefixes everywhere; 62 minutes | outage 21:52-22:54 UTC, 62 minutes; config error "remained dormant in the production network" until a second change "triggering a global refresh of network configuration" (search-verified) |
| 10 | Salesforce | Multi-Instance Service Disruption on May 11-12, 2021 | postmortem | 2021-05 | 2026-09-04 | https://help.salesforce.com/s/articleView?id=000390251&language=en_US&type=1 | A DNS change shipped through the emergency process without qualifying as an emergency; the script misbehaved during a zone transfer and servers did not restart | the change "avoided the extensive review scrutiny" of the standard process; "there was no active or imminent Severity-0, Severity-1 or Severity-2 incident, so the EBF process should not have been used"; ~4 hours down (search-verified; corroborated by [22]) |
| 11 | Roblox | Roblox Return to Service 10/28-10/31 2021 | postmortem | 2022-01 | 2026-09-04 | https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ | The service-discovery layer (Consul) was the single point: a new streaming feature plus a BoltDB pathology under load; one cluster served every workload; 73 hours | "Enabling a relatively new streaming feature on Consul under unusually high read and write load led to excessive contention"; "A single Consul cluster supporting multiple workloads exacerbated the impact"; 73 hours, 100% restored 16:45 Oct 31 (search-verified) |
| 12 | Kubernetes SIG-Network | KEP-1024: NodeLocal DNS Cache | adr | 2019 (GA 1.18) | 2026-09-04 | https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1024-nodelocal-cache-dns | The accepted fix for cluster DNS flakiness was a per-node cache that bypasses conntrack and upgrades upstream queries to TCP | "Skipping iptables DNAT and connection tracking will help reduce conntrack races and avoid UDP DNS entries filling up conntrack table"; "DNS timeouts usually up to 30s (3 retries x (5s timeout for A + 5s timeout for AAAA))" (fetched directly) |
| 13 | IETF | RFC 8767: Serving Stale Data to Improve DNS Resiliency | adr | 2020-03 | 2026-09-04 | https://datatracker.ietf.org/doc/html/rfc8767 | The standards body's answer to resolution outages is to serve expired records when the authority is unreachable, capped at 7 days; Akamai ran it since 2011 | updates TTL definitions "so that data can be kept in the cache beyond the TTL expiry", "suggests a cap of 7 days"; the algorithm "has been in use on Akamai's production network since 2011" (search-verified) |
| 14 | gRPC | doc/naming.md, gRPC Name Resolution | adr | maintained | 2026-09-04 | https://github.com/grpc/grpc/blob/master/doc/naming.md | DNS is the default name system but explicitly a pluggable one; production deployments swap in xDS or registry-backed resolvers | "gRPC supports DNS as the default name-system. A number of alternative name-systems are used in various deployments." (fetched directly) |
| 15 | Kubernetes | Issue #56903: DNS intermittent delays of 5s | source | 2017-12 | 2026-09-04 | https://github.com/kubernetes/kubernetes/issues/56903 | The 5-second DNS delay was one of the most-hit issues in Kubernetes, rooted in kernel conntrack races on parallel UDP queries | "DNS lookup is sometimes taking 5 seconds" (issue title/body); root cause traced to conntrack races (search-verified; corroborated by [12], [17]) |
| 16 | Weaveworks | weave issue #3287: DNS lookup timeouts due to races in conntrack | source | 2018 | 2026-09-04 | https://github.com/weaveworks/weave/issues/3287 | The same race independently diagnosed in the Weave network layer; the kernel, not any DNS server, drops the packet | issue title: "DNS lookup timeouts due to races in conntrack" (search-verified) |
| 17 | Martynas Pumputis | Racy conntrack and DNS lookup timeouts | blog | 2018-08-16 | 2026-09-04 | https://lambda.lt/blog/2018/racy_conntrack.html | The author's write-up of the kernel races behind [15]/[16]: parallel A and AAAA over the same socket race in conntrack insertion, one packet is dropped, resolver waits out its 5s timeout | "Racy conntrack and DNS lookup timeouts", 2018-08-16 (search-verified; author of the kernel fixes) |
| 18 | CoreDNS | Issue #4309: Add support for serving stale data according to RFC 8767 | source | 2020-11 | 2026-09-04 | https://github.com/coredns/coredns/issues/4309 | Serve-stale demand reached cluster DNS; CoreDNS added serve_stale to the cache plugin |
issue title: "[Cache] Add support for serving stale data according to RFC 8767" (search-verified) |
| 19 | musl | musl 1.2.4 released (TCP fallback) | source | 2023-05-02 | 2026-09-04 | https://www.openwall.com/lists/musl/2023/05/02/1 | After a decade of refusals, musl added DNS TCP fallback, ending truncation failures that hit every Alpine container | "adds TCP fallback to the DNS stub resolver, fixing the longstanding inability to query large DNS records" (search-verified) |
| 20 | musl | Thread: TCP support in the stub resolver (rejected approach) | source | 2020-04-21 | 2026-09-04 | https://www.openwall.com/lists/musl/2020/04/21/8 | The recorded argument for NOT doing TCP fallback: truncated results were considered acceptable, bounded, and cheaper; reversed in 1.2.4 | maintainer position: truncated results "not just acceptable but better ux ... you save major round-trip delays" (position search-verified across list thread and author's public statements) |
| 21 | APNIC / Geoff Huston | ISP Column: notes from DNS-OARC 36 | blog | 2021-12 | 2026-09-04 | https://www.potaroo.net/ispcol/2021-12/oarc36.html | Independent analysis of the Slack incident mechanism: the NSEC bitmap error and aggressive negative caching per RFC 8198 | "an error in the NSEC type bitmap produced by the Route 53 server for wildcard domains" (search-verified) |
| 22 | The Register | That Salesforce outage: Global DNS downfall started by one engineer trying a quick fix | blog | 2021-05-19 | 2026-09-04 | https://forums.theregister.com/forum/all/2021/05/19/salesforce_root_cause/ | Press corroboration of the Salesforce RCA: EBF process subverted, DNS servers failed to restart | "domain name servers that did not restart as expected after a configuration change" (search-verified) |
| 23 | Netflix | Eureka wiki: Eureka at a glance | source | maintained | 2026-09-04 | https://github.com/Netflix/eureka/wiki/Eureka-at-a-glance | Netflix's stated reason for a registry over DNS: DNS routes traffic "to servers which may not be healthy or may not even exist"; client-side cache survives registry outage | "your application can be resilient to the outages of the load balancers, since the information regarding the available servers is cached on the client"; heartbeats every 30s, eviction ~90s, clients poll every 30s (fetched directly) |
| 24 | Netflix | Netflix Shares Cloud Load Balancing And Failover Tool: Eureka! | blog | 2012-09 | 2026-09-04 | http://techblog.netflix.com/2012/09/eureka.html | Eureka announced publicly as the mid-tier discovery layer in 2012; the choice predates cloud DNS maturity and was framed around failover | "AWS Service registry for resilient mid-tier load balancing and failover" (repo tagline, fetched directly from [25]) |
| 25 | Netflix | Netflix/eureka repository | source | maintained | 2026-09-04 | https://github.com/Netflix/eureka | The registry is still maintained and framed as discovery + failover, not naming | "Eureka is a RESTful ... service that is primarily used in the AWS cloud for the purpose of discovery, load balancing and failover of middle-tier servers" (fetched directly) |
| 26 | Stripe | The secret life of DNS packets: investigating complex networks | blog | 2019 | 2026-09-04 | https://stripe.com/blog/secret-life-of-dns | Stripe's resolution path is layered caches: Unbound on every host, forwarding to an Unbound fleet with per-destination forwarding rules and per-rule timeout isolation | Stripe "operates a cluster of DNS servers running Unbound ... configured to forward DNS queries to different upstream destinations based on the domain"; Unbound also "running locally on every host" (search-verified) |
| 27 | Marco Pracucci | Kubernetes pods /etc/resolv.conf ndots:5 option and why it may negatively affect your application performances | blog | 2019 | 2026-09-04 | https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html | ndots:5 turns every external lookup from a pod into a series of search-domain expansions before the real query is tried | title claim; ndots:5 default documented in the post (search-verified) |
| 28 | Jung, Sit, Balakrishnan, Morris (MIT) | DNS Performance and the Effectiveness of Caching | paper | 2001 | 2026-09-04 | https://conferences.sigcomm.org/imc/2001/imw2001-papers/89.pdf | Low A-record TTLs barely hurt hit rates; the protective caching is NS-record caching; and failure traffic dominates packets | "reducing the TTLs of address (A) records to as low as a few hundred seconds has little adverse effect on hit rates"; "In the most recent MIT trace, 23% of lookups receive no answer" (search-verified) |
| 29 | Moura, Heidemann, Hardaker, Schmidt | Cache Me If You Can: Effects of DNS Time-to-Live (IMC '19) | paper | 2019-10 | 2026-09-04 | https://dl.acm.org/doi/10.1145/3355369.3355568 | TTL is a latency lever: raising a ccTLD's TTL cut median latency from 183 ms to 28.7 ms | "longer TTLs have significant promise in reducing latency, reducing it from 183ms to 28.7 ms" for .uy (search-verified) |
| 30 | Moura et al. | When the Dike Breaks: Dissecting DNS Defenses During DDoS (IMC '18) | paper | 2018-10 | 2026-09-04 | https://dl.acm.org/doi/10.1145/3278532.3278534 | Full caches let about half of clients ride out an authority outage shorter than the TTL; client retries multiply traffic on the struggling authority up to 8x | "when caches are full they allow about half of clients to ride out server outages"; "retries during DDoS attacks increase normal traffic up to 8x" (search-verified) |
| 31 | USENIX SREcon22 EMEA | Slack's DNSSEC Rollout: Third Time's the Outage (talk, R. Elvira et al.) | talk | 2022-10 | 2026-09-04 | https://www.usenix.org/conference/srecon22emea/presentation/elvira | Conference account of [7]/[8]; the title itself records that two aborted attempts preceded the outage | title: "Third Time's the Outage"; slides at https://www.usenix.org/system/files/srecon22_slides_elvira.pdf (search-verified) |
| 32 | RIPE 77 | When the Dike Breaks presentation (Moura) | talk | 2018-10 | 2026-09-04 | https://ripe77.ripe.net/presentations/11-presentation.pdf | Slide deck of [30] presented to operators; caching as the DNS's principal DDoS defense | slide deck of the IMC '18 paper (search-verified) |
| 33 | AWS | Amazon Route 53 Service Level Agreement | vendor | current | 2026-09-04 | https://aws.amazon.com/route53/sla/ | The DNS data plane is the only AWS service sold with a 100% availability SLA; the SLA covers answering queries, not changing records | "100%" Monthly Uptime Percentage for Route 53 (hosted zone query serving); resolver endpoints carry 99.99% (search-verified) |
| 34 | AWS | Well-Architected REL11-BP04: Rely on the data plane and not the control plane during recovery | vendor | 2025-02 | 2026-09-04 | https://docs.aws.amazon.com/wellarchitected/2025-02-25/framework/rel_withstand_component_failures_avoid_control_plane.html | AWS's own guidance splits Route 53 into a us-east-1 control plane and a globally distributed data plane, and tells customers to fail over via health checks (data plane), never via record updates (control plane) | "Route 53's data plane is "statically stable in the face of ... a control plane failure"; control plane "is hosted in us-east-1"; data plane "runs across over 200 PoP locations" (search-verified) |
| 35 | Internet Society | Lessons from the Cloudflare 1.1.1.1 Outage: A Resilience Perspective | blog | 2025-07 | 2026-09-04 | https://pulse.internetsociety.org/blog/lessons-from-the-cloudflare-1-1-1-1-outage-a-resilience-perspective | Scale of the failed resolver, and the resilience reading: a single upstream resolver is a single point of failure | "As of early 2025, 1.1.1.1 handles approximately 1.9 trillion DNS queries daily"; "depending on a single upstream DNS resolver ... introduces a single point of failure" (search-verified) |
| 36 | Data Center Frontier | Metaverse Platform Roblox Adds Data Center to Address 73-Hour Outage | blog | 2022 | 2026-09-04 | https://www.datacenterfrontier.com/featured/article/11427647/metaverse-platform-roblox-adds-data-center-to-address-73-hour-outage | Cost of the discovery-layer failure: about $25M in lost bookings, per the company | "The downtime cost Roblox an estimated $25 million in lost bookings, the company said" (search-verified) |
| 37 | Kentik | Facebook's historic outage, explained | blog | 2021-10 | 2026-09-04 | https://www.kentik.com/blog/facebooks-historic-outage-explained/ | External measurement view of [4]: the world saw BGP withdrawals of the nameserver prefixes minutes before DNS died | external BGP monitoring saw "a large number of BGP changes (mostly route withdrawals) for Facebook's ASN" about five minutes before DNS failed (search-verified) |
| 38 | ThousandEyes | AWS Outage Analysis: October 20, 2025 | vendor | 2025-10 | 2026-09-04 | https://www.thousandeyes.com/blog/aws-outage-analysis-october-20-2025 | Independent external timeline of [1]-[3], including the long dependent-service tail after the DNS record was fixed | independent monitoring corroborates the ~2:40 AM DNS repair and the multi-hour recovery tail (search-verified) |
| 39 | musl / R. Felker | dns: implement tcp fallback in __res_msend query core (commit) | source | 2023-04 | 2026-09-04 | https://git.musl-libc.org/cgit/musl/commit/?id=51d4669fb97782f6a66606da852b5afd49a08001 | The actual commit reversing the decade-old position in [20] | commit title: "dns: implement tcp fallback in __res_msend query core" (search-verified) |
Tier mix (target: mostly tiers 5+)
postmortem 11 rows / 6 incidents · adr 3 · source 8 · paper 3 · talk 2 · blog 9 · vendor 3 Organisations: AWS, Meta, Slack, Cloudflare, Salesforce, Roblox, Kubernetes/CNCF, IETF, gRPC, Weaveworks, CoreDNS, musl, Netflix, Stripe, APNIC, MIT, USC/ISI+SIDN Labs, USENIX, RIPE, Internet Society, Kentik, ThousandEyes, Data Center Frontier, The Register. Hosts: 24 distinct.