A globally distributed API experiences a regional networking failure. How should DNS, health checks, traffic steering, connection draining, failover and consistency interact?

Why DNS is the wrong primary failover mechanism

DNS TTLs are advisory. Resolvers, operating systems, browsers and application runtimes all cache independently and frequently ignore short TTLs. A 60-second TTL routinely means twenty minutes of residual traffic to a dead region, and some clients cache for the lifetime of the process.

DNS is a good mechanism for planned changes and a poor one for failure response.

What actually works

• Anycast, where the same address is announced from many locations and the network routes to the nearest healthy one. Withdrawal of a route removes a region in seconds without touching DNS. This is the primary mechanism for edge-terminated traffic.
• Health checks that test the real path, not a static endpoint. A /health returning 200 while the database is unreachable is worse than no check, because it keeps a broken region in rotation.
• Client-side awareness for anything you control — a mobile app or SDK that tries an alternative endpoint on failure recovers faster than any server-side mechanism.
• Short-lived connections or explicit connection draining, since long-lived connections survive a routing change and continue sending traffic to a failed region until they are closed.

The interaction with consistency

The routing layer can move traffic in seconds. The data layer usually cannot, and that mismatch is where regional failover goes wrong: traffic arrives in a healthy region whose replica is behind, or — worse — whose writes now conflict with the failed region's.

Route reads freely and route writes deliberately. Reads can go anywhere with a staleness bound. Writes should go to the entity's home, and if the home is gone, the decision to promote elsewhere must be quorum-based or human, because an automatic promotion that is wrong produces two writers and a split history.

The failure most teams have

The failover path is exercised for the first time during the incident. Regional failover involves DNS, certificates, identity, network paths, capacity in the surviving region, and downstream allow-lists — and any one of them being wrong makes the failover fail. Regularly withdrawing a region deliberately is the only way to know it works.