A 43-second network blip triggers automated database failover across regions. Service is degraded for over 24 hours. Analyse.

What the interviewer is testing

Whether you recognise that the outage duration was determined by the reconciliation, not by the trigger.

The chain

A brief loss of connectivity made the primary unreachable from the failover system's vantage point. Automation, working exactly as configured, promoted a remote replica. During the window both locations accepted writes. When connectivity returned, the two had diverged.

Reconciling divergent writes across a high-volume database is a bespoke, careful, manual operation. There is no tool that does it and no way to rush it without losing customer data — which is why 43 seconds became 24 hours.

The design failures

A failover threshold shorter than the duration of transient network events. Brief partitions are far more common than genuine host failure, so a 43-second threshold means failing over for every blip.

No fencing. The original primary continued accepting writes after being deposed. A failover that does not positively prevent the old primary from writing has not completed; it has created two primaries.

Automatic cross-region promotion. The blast radius of an incorrect decision is far higher across regions than within one, and the recovery is far harder.

The changes

Lengthen the threshold so it exceeds observed transient events, accepting slower failover in exchange for not failing over unnecessarily.

Fence at the storage layer using increasing tokens, so a deposed primary's writes are rejected by the resource rather than depending on it knowing it was deposed.

Automatic within a region, deliberate across regions. Many organisations adopt exactly this after an incident of this shape.

Rehearse the reconciliation, because if divergence does occur, the procedure should not be invented during the incident.

What a strong answer adds

The principle: the cost of a wrong failover is usually far higher than the cost of a slow one. Most failover configurations are tuned to minimise detection time, optimising the wrong variable — availability during a genuine failure is worth less than avoiding split brain, because one is recoverable in minutes and the other in days.

Common weak answers

Faster health checks. More replicas, which does not address the two-writer problem.