A team designs active-active across two regions with automatic failover based on health checks. What is wrong?
Show the full answer Hide the answer
What the interviewer is testing
Whether you recognise that two nodes cannot form a majority, which is the most common structural flaw in two-region designs.
What is wrong
A partition between the two regions is indistinguishable from the other region failing. Each side sees the other as unreachable, each health check passes locally, and each concludes it should take over.
Both accept writes. That is split-brain, and it is worse than an outage because it does not announce itself: both halves appear healthy, both serve traffic, and the damage surfaces when the partition heals and two divergent histories cannot be merged.
With two participants there is no majority. Any quorum rule that one side can satisfy, the other can satisfy too.
The fixes
A third site as a witness or arbiter. It holds no data and participates in the quorum, so a partition produces exactly one majority. This is the standard answer and it is cheap — a small instance in a third region.
Single-writer with a manual, deliberate failover. One region is authoritative; failover requires a human decision and a positive action that fences the old primary so it cannot resume writing. Slower to recover and correct.
Regional partitioning. Each region owns a subset of the data as the sole writer. There is no contested leadership, because no record has two possible owners. This is often the best answer for multi-region systems and is under-used.
The operational rule that must accompany any of these
The minority side must refuse writes even though it is running perfectly well, and that refusal is the design working as intended. It is also the part that gets overridden under pressure by an operator who sees a healthy region declining traffic — so the reasoning needs to be in the runbook, not just in the design.
What a strong answer adds
Fencing at the storage layer: a deposed primary that wakes up and writes must be rejected by the resource, not merely told it is no longer leader. Without that, a paused process resuming after a failover is a data corruption path that no timeout prevents.
Common weak answers
Adding more health checks, which cannot distinguish partition from failure. Faster failover, which increases the split-brain risk.