A leader-based database cluster loses network connectivity between the leader and its followers, but the leader is still running and still accepting writes from application servers that can reach it. What happens, and how does correct leader election prevent it?

The failure

The followers cannot reach the leader, conclude it has failed, and elect a new one. The old leader has not failed — it simply cannot see them — and it continues accepting writes from application servers on its side of the partition.

Now two nodes believe they are the leader. Both accept writes. Both are internally consistent. The data diverges, and when the partition heals there is no correct merge, because both sets of writes were acknowledged to clients as durable. This is split-brain, and it is the canonical way distributed databases lose data permanently.

Note the asymmetry that makes it dangerous: it is not detected at the moment it occurs. It is detected later, during reconciliation, when someone must choose which acknowledged writes to discard.

Why timeouts alone cannot prevent it

A node cannot distinguish "the leader has crashed" from "I cannot reach the leader". This is the fundamental limit — no timeout value fixes it, because the information required is not available. Any protocol relying on failure detection alone is guessing.

The three mechanisms that actually prevent it

1. Quorum. A leader is only legitimate with acknowledgement from a majority. In a partition, at most one side can hold a majority, so at most one leader can exist. The minority side cannot elect a leader and cannot commit writes. This is why cluster sizes are odd, and why a two-node cluster cannot be made safe by cleverness.

2. Leases with bounded clock error. A leader holds a time-limited lease and must stop accepting writes before it expires unless renewed. A new leader waits out the old lease before starting. This turns "I think the leader is dead" into "the old leader has provably stopped", but it depends on bounded clock drift, which is an assumption that deserves monitoring rather than faith.

3. Fencing tokens. Every leadership term carries a monotonically increasing number. Downstream systems — storage, log, external services — record the highest token they have seen and reject anything from a lower term. This is the crucial mechanism, because it protects against the case the other two miss: an old leader that was paused (a long garbage-collection pause, a suspended VM, a blocked disk) and wakes up believing its lease is still valid. It cannot do damage, because storage refuses its writes.

Where real systems still get hurt

• Fencing implemented in the coordinator but not the data path. The lock service is correct; the storage layer accepts whatever it is sent. The token must be checked where the effect happens.
• Two-node clusters plus a manual tiebreak. Under pressure, the human picks wrong.
• Asymmetric partitions. A can reach B but B cannot reach A. Failure detection based on one-way reachability produces flapping leadership.
• Application servers cached onto the old leader. Even with correct election, clients holding connections to the demoted node keep writing until something refuses them — which is, again, fencing.

The lesson

Leader election is not about choosing a leader. It is about guaranteeing that the previous leader cannot act. Systems that treat it as an election problem rather than a revocation problem lose data in exactly this scenario.