Failover Propagation Delay

A failover runbook measures the control plane: promote the replica, update the record, shift the traffic policy. Ninety seconds, verified, repeatable.

The recovery time a user experiences is that number plus the time for every client to notice, and clients notice at their own pace, governed by caches and pools that nobody configured with a failover in mind. Twenty-five minutes of user-visible outage after a ninety-second failover is an ordinary result, and it is entirely accounted for by this delay.

Why it matters

An organisation that publishes the control-plane number as its RTO has promised something it has never achieved. The gap is invisible in drills that restart everything, because a freshly started process has no stale cache and no pooled connection to a dead host. The delay only appears with long-running processes, which is what production consists of.

Implementation patterns

• Short TTLs on failover-relevant records, 30 to 60 seconds, and verified runtime DNS caching settings. A TTL is a lower bound that resolvers, containers and language runtimes each ignore in their own way; some runtimes have historically cached name lookups for the life of the process.
• Bounded maximum connection age in every pool, a few minutes, so no pool can outlive a topology change by more than that. This costs a little connection churn and removes an entire failure class.
• A stable endpoint that clients re-resolve - a virtual address, a proxy layer, a service-discovery entry - so failover does not depend on each client's caching behaviour.
• Circuit breaker reset intervals that are shorter than your target recovery time, otherwise breakers correctly keep a recovered dependency out for minutes.
• Region-scoped assets listed in the plan: credentials, encryption keys, certificates, allowlisted source addresses, licence servers. These are configured once and forgotten, and they are what turns a database failover into a multi-team incident.
• Measure from outside: a synthetic transaction from a client network, and the RTO is the time until it succeeds again.

Industry example

The mechanism is visible in GitHub's October 2018 incident, where a 43-second network partition triggered an automated cross-coast database failover and the consequences ran for about 24 hours while writes on both sides were reconciled. The failover itself was quick; everything expensive happened downstream of it. The general property holds at every scale: the control-plane action is the beginning of recovery, not the end of it, and the parts that take time are the ones involving state held by other components.

Failure scenarios

• A runtime caching DNS forever, so a long-running service never fails over until it is restarted.
• Connection pools holding dead sockets, which fail one at a time as reads time out rather than all at once.
• Breakers open with a long reset, keeping a recovered dependency out.
• Caches warmed against the old region, so the new one serves correctly and slowly, which presents as a partial outage rather than a recovery.
• A standby scaled down to save cost, which must scale up under full load precisely when the cloud's capacity is in demand.
• Region-scoped credentials or keys nobody mentioned, producing authorisation errors that look like an application bug.

Trade-offs

Every mitigation costs something in steady state. Short TTLs mean more DNS queries and a resolver dependency on the hot path. Bounded connection ages mean continuous reconnection churn and a small latency cost on the requests that pay for a new connection. A proxy layer that gives a stable endpoint is another hop and another thing to operate.

These are small, continuous costs bought against a rare, large one, and they are a good trade for any system with a published RTO. They are a poor trade for a system whose recovery time nobody has committed to, where the honest answer is to leave the defaults and document the real number.

When not to use it

Do not engineer client convergence for a system that can be down for an hour without consequence. Measure it first: run a failover, watch from outside, and find out whether the delay is 2 minutes or 25. Most teams have never measured it, and the measurement costs one afternoon while the mitigations cost a quarter.

Interview question

Q: Your documented RTO is 15 minutes, and a drill showed the database failover completing in 90 seconds. A real failover took 40 minutes to restore service. Explain the gap, and tell me what you would change and what you would publish.

What a strong answer covers: that control-plane time is a component measurement and client convergence is the rest · DNS caching at several layers including runtime defaults · pooled connections to dead hosts failing one timeout at a time · breakers with long resets, cold caches and region-scoped credentials · bounded connection age and short TTLs as the concrete changes · measuring RTO with an external synthetic transaction · and publishing the measured number rather than the runbook's.

Quick check

Quiz: Why does restarting a service fix a failover problem instantly? Because the stale state lives in the process: cached name resolution, pooled connections and open breakers, none of which a restart preserves.

Flashcard: What is the real RTO? Time from failure until a synthetic transaction from outside succeeds again, not the time the control-plane operation took.