A regional failover completes in 90 seconds per the runbook - the database is promoted and traffic is redirected - but the application stays broken for 25 minutes. The new region's services are healthy and idle. Where is the time going?

The first three things to look at

1. Who is still talking to the old region. Check inbound connection counts at the failed region before assuming nothing points there. The most common answer is that plenty still does.
2. Whether the errors are connection failures or authorisation failures. Refused connections point at addressing; permission errors point at region-scoped credentials or key material.
3. Whether the clients recovering are new processes or existing ones. If restarting a service fixes it instantly, the fault is per-process cached state, which narrows it to two or three causes.

The diagnosis

The runbook measured the control-plane operation. The recovery time a user experiences is the control-plane time plus the time for every client in the system to notice, and clients are where the minutes are.

• DNS caching at several layers. A record's TTL is a lower bound, not a guarantee: resolvers, containers and application runtimes each cache independently. Some runtimes historically cached name lookups for the process lifetime by default, which means the process never notices the failover at all.
• Connection pools holding open sockets to old addresses. A pooled connection to a host that is gone does not fail until something times out, and a pool configured with no maximum connection age will hold a dead peer until a read times out on each one in turn.
• Circuit breakers in the open state with a long reset interval, correctly protecting against a dependency that recovered two minutes ago.
• Region-scoped credentials or encryption keys that the failover procedure never mentioned, because they were configured once and are not part of the database story.
• Caches warmed against the old region, so the new region serves correctly and slowly, which looks like a partial outage.

Each of these is a few minutes. Together they are 25.

The fix, in order

1. Short TTLs on failover-relevant records - 30 to 60 seconds - and verified runtime DNS caching settings. Verified, because the default in some runtimes is effectively infinite and no one discovers that during a drill that restarts everything.
2. Bounded connection lifetime in every pool, so a pool cannot outlive a topology change by more than that bound. A maximum age of a few minutes costs a little churn and removes an entire failure class.
3. Failover as a client-aware operation. The procedure must include what clients do, not only what the database does, and the drill must exercise long-running processes rather than freshly started ones.
4. Prefer a stable endpoint over a changing address: a virtual address, a proxy layer or a service-discovery entry that clients re-resolve, so failover does not depend on every client's caching behaviour.

The measurement that would have prevented the surprise

Measure RTO from the client's perspective: time from failure to the point where a synthetic transaction from outside succeeds again. The runbook's 90 seconds is a component measurement, and publishing it as the RTO is how an organisation ends up promising a recovery time it has never achieved.

When this is the wrong diagnosis

If nothing recovers even after restarts, stop looking at clients: it is capacity or configuration in the new region, and the usual cause is that standby capacity was scaled down to save money and now has to scale up under full load. That failure is prevented by static stability - provisioning the standby for the traffic it would take - rather than by any client-side change.