You use DNS with a 60-second TTL for regional failover. During a test, some traffic still hit the failed region 40 minutes later. Explain.

What the interviewer is testing

Whether you know that DNS TTLs are advisory in practice, and can propose a mechanism that is not.

The explanation

TTLs are not reliably honoured. Three layers ignore them:

Intermediate resolvers sometimes enforce their own minimum TTLs, particularly on some ISP and corporate networks.

Application runtimes cache DNS results independently. The JVM historically cached indefinitely by default, and many HTTP clients and connection pools resolve once at startup and hold the address for the process lifetime. This is the largest contributor and it is invisible from the DNS side.

Existing connections do not re-resolve at all. A client with a live keep-alive connection to the failed region continues using it until it is closed.

So DNS failover is eventual and partial: most traffic moves within a few multiples of the TTL, and a residual fraction persists far longer. Any plan requiring the old endpoint to stop receiving traffic at a specific moment is unsound.

Better mechanisms

Anycast — the same address announced from multiple locations, with failover by withdrawing an announcement. Handled by the routing layer within seconds, with no dependence on client caching.

A global load balancer with health checks, where the endpoint address is stable and the balancer redirects behind it.

Client-side awareness of multiple endpoints, with health checking and failover in the client library. Effective and requires control of the clients.

What to do with DNS regardless

Keep TTLs short on records you may need to move, and lower them well in advance of a planned migration — lowering a TTL only takes effect after the previous, longer TTL expires everywhere, which is the step forgotten the day before a cutover.

What a strong answer adds

The design consequence: the failed region must continue to fail safely for the residual traffic rather than being torn down. Removing it entirely turns a partial failover into connection errors for that fraction of users, sometimes for a long time.

Common weak answers

Reducing the TTL further. Concluding the test was flawed.