After a DNS failover, about 8% of clients keep failing for roughly 30 minutes. A dig against the authoritative servers and against the public resolvers returns the new record immediately, and the record's TTL is 60 seconds. What is holding those clients?

The first three things to look at, in order

1. What the failing clients are resolving, and what answer they get. Not what the new record says - what the client's own resolver returns, from inside the failing network.
2. Whether a negative answer was cached. If the failover created the record rather than changing it, resolvers that queried during the gap cached an NXDOMAIN, and the lifetime of that cached negative answer is not governed by the record's TTL.
3. The zone's SOA minimum field, which is what governs negative caching lifetime, together with the resolver's own cap.

The diagnosis

Negative caching is the mechanism, and the SOA record is the configuration. A resolver that receives "no such name" caches the absence, for a duration taken from the zone's SOA minimum value (bounded by the resolver's own limits). A zone with an SOA minimum of 1800 holds that negative answer for 30 minutes, whatever the eventual record's 60-second TTL says, because the negative answer was cached before the record existed.

That explains the shape of the symptom exactly: a minority of clients, those whose resolver happened to query during the gap, failing for a fixed period unrelated to the record TTL, while every manual check succeeds. Manual checks query a resolver that did not cache the negative answer, or query authoritative servers directly - which is why the diagnostic that would have found this in two minutes is to query the failing client's own resolver, not the authority.

The misleading clue is the 60-second TTL. It is correct, it is being honoured, and it has nothing to do with this failure.

The fix, in order of deployability

1. Lower the SOA minimum to something commensurate with your failover expectations - 60 to 300 seconds - accepting more query load for absent names.
2. Stop using record creation as a failover mechanism. Have the record always exist and change its value: a positive answer with a short TTL has a bounded, controllable lifetime. Failover by CNAME swap or by health-checked record value, never by creating a name that did not exist.
3. Move failover below DNS where the recovery target is tight. Anycast with health-checked withdrawal, or a load balancer with a stable name, removes resolver caching from the recovery path entirely.
4. Pre-create every name you might need, including regional and failover names, pointing at something harmless until needed.

When this is the wrong answer

Negative caching is not a quirk; it is specified behaviour, defined in RFC 2308 (1998) with the SOA minimum field as its governing value, and it exists to protect authoritative servers from floods of queries for names that do not exist. Lowering the SOA minimum therefore buys failover speed and pays in query load - typically a modest cost, and not zero for a zone that attracts junk queries. For a planned migration with hours of notice, DNS is a perfectly good control plane and none of this machinery is needed: create the record, wait out the propagation, then switch. The caution is about treating DNS as a fast failover mechanism, which it is not - published resolvers and client stacks honour TTLs inconsistently, some enforce their own floors, and long-lived client processes cache in-process for the life of the connection pool. Design for a recovery objective of minutes if DNS is in the path; if you need seconds, the mechanism has to be routing or a stable endpoint, and no amount of TTL tuning changes that.