Failover

Definition

Failover promotes a standby to primary. The mechanics are usually straightforward; the difficulty is in deciding when, in preventing the old primary from continuing to act, and in getting back afterwards.

The three hard parts

1. Detection without false positives. A network partition looks exactly like a dead primary from the standby's perspective. Failing over unnecessarily is often worse than the fault — you have caused an outage to fix one that did not exist. Requirements: multiple independent observers, a quorum before acting, and a deliberate delay so transient blips do not trigger it.

2. Fencing. The old primary must be prevented from continuing to write, not merely asked to stop. A primary that is unreachable to the cluster may still be reachable to clients, and if both accept writes you have split brain and divergent histories that cannot be automatically merged. Mechanisms: quorum so the isolated side cannot achieve one, fencing tokens the storage layer checks and rejects, or forcibly powering the old primary off.

3. Failback. Returning to the original primary, reconciling anything written during the outage, and re-establishing replication. Harder than the failover and almost never rehearsed.

Automatic or manual

Automate when detection is reliable, the failure is common, and the cost of a false positive is low — a stateless instance behind a load balancer, for example.

Keep a human in the loop when the consequence of a wrong decision is data divergence, when the failure is rare enough that the automation will not be exercised, or when the correct action depends on context the automation cannot see. Database primary failover across regions frequently belongs in this category, and there is no shame in it — but the manual path must be practised, documented and fast, or the RTO is fiction.

The worst option is automation nobody trusts, which is disabled during the incident by someone who then performs an unrehearsed manual procedure under pressure.

What must be true beforehand

• Replication lag monitored, so failover does not land on a badly stale replica.
• Clients that reconnect and retry cleanly, since every failover severs connections.
• Idempotent writes, because in-flight requests will be retried across the switch.
• DNS and connection caching bounded, or traffic continues to the old endpoint.
• The runbook accessible from outside the failed system.

Failure scenarios

• Split brain from missing fencing.
• Flapping, where a marginal network causes repeated failovers, each severing connections.
• Failover onto a stale replica, silently losing acknowledged writes.
• The standby lacks capacity for full production load.
• Automation never tested, so its first real execution is also its debut.

Interview question

"Your primary is unreachable from the monitoring system but still serving some clients. Do you fail over? What do you need in place to make that safe?"