On 21 October 2016 a managed DNS provider was hit by three waves of distributed denial-of-service traffic beginning about 11:10 UTC, which the provider and Flashpoint confirmed involved a Mirai botnet of compromised consumer devices. Well-known sites that used it as their only authoritative DNS were unreachable for much of that day although their own servers were healthy throughout. What failed, which design decision made it possible, and what would you have changed?

The trigger

Volumetric attack traffic from a large botnet of consumer devices, aimed at the authoritative name servers of a managed DNS provider. The attack arrived in waves through the day; the first targeted infrastructure in the north-eastern United States and later waves were broader.

Why it propagated

Authoritative DNS is a hard dependency that fails closed. If no name server answers for your zone, your service does not degrade - it ceases to exist for any client that does not already hold a cached answer. Application servers, databases and CDNs were all healthy and all unreachable, because nothing could resolve the names.

Two mechanisms made the blast radius larger than the attack:

• Recursive resolvers retry and then give up. Once a resolver's cached record expires it must re-resolve; under attack that query fails, and the client gets SERVFAIL or a timeout rather than the last known good answer. Short TTLs, adopted for fast failover, shortened the grace period.
• A single provider was the whole answer. The NS record set for each affected zone named one organisation's name servers, so there was no second path for a resolver to try.

Why detection did not help

Every monitoring signal inside the affected companies was green: their own infrastructure was fine. The only failing signal was external resolution, which most teams did not monitor from outside their own networks at the time. The incident was detected by customers.

The structural fix versus the tempting local fix

The tempting fix is to lower TTLs further, or to move provider. Neither addresses the dependency: a lower TTL makes you more exposed to an authoritative outage, and a different single provider is the same design.

The structural fix is two authoritative DNS providers serving the same zone, with the NS record set naming both, so a resolver that cannot reach one uses the other. Mechanically:

1. Either zone transfer (AXFR/IXFR) from a primary to the second provider, or a dual-push pipeline that applies every change to both through your own automation. Dual push avoids depending on one provider's transfer implementation, at the cost of owning the consistency problem.
2. Verify both answer identically, continuously. A second provider with a stale zone is worse than none, because resolvers will use it.
3. Keep the record types you use inside the intersection of what both support. This is the real blocker in practice: provider-specific features - ALIAS or ANAME records at the apex, weighted or geo-steered answers, health-checked failover - frequently do not port, and a zone built on them cannot be served by two providers without redesign.
4. Monitor resolution from outside, from multiple networks, as a first-class signal.

Cost is modest - a second provider's bill for the same zone is typically tens to low hundreds of dollars a month at ordinary query volumes - and the design cost is the feature intersection above.

The general lesson

Enumerate the dependencies that fail closed and that you do not operate. Authoritative DNS, certificate issuance, an identity provider and an egress NAT path are the usual list. For each, the question is not "how reliable is the vendor" but "what is the second path, and has anyone tried it".

When this is the wrong fix

A single provider is defensible for an internal zone whose clients are all inside a network you operate, where a resolution outage and an application outage are the same event anyway. It is also defensible - briefly - where the business genuinely depends on health-checked latency-aware steering at the apex that only one provider offers. In that case be explicit: record it as a known single point of failure, keep an exercised runbook for moving the zone, and do not claim diversity you do not have. What is never defensible is lowering the TTL to compensate, which increases exposure to exactly this failure.