A leader-based coordination service loses contact with its followers while the leader itself remains healthy and continues serving. What failure modes emerge, and what prevents split-brain?

The failure

The old leader believes it is still the leader. The followers elect a new one. Two leaders now accept writes, and both are behaving correctly according to what they can observe. Nothing detects the problem until the partition heals and the two histories have to be merged — which, for anything with financial or ordering semantics, they cannot be.

What prevents it

• Quorum on both sides of the operation. A leader must confirm it still holds a majority before acknowledging a write, not merely at election time. A leader that cannot reach a majority must stop serving, even though it is perfectly healthy — the willingness to stop is the entire mechanism.
• Leases with a bound on clock error. A leader holds leadership for a bounded period and must renew. If it cannot renew, it steps down before the lease expires, and the new leader waits out the old lease before acting. This depends on bounded clock drift, which is an assumption that must be monitored and not merely hoped for.
• Fencing tokens. Each leadership term gets a monotonically increasing number, passed to every downstream system, which rejects anything carrying an older token. This is what protects the resources the leader acts on, and it is the part most commonly omitted: a lease stops the old leader from believing it is leader, a fencing token stops its already-issued write from landing.
• An odd cluster size across independent failure domains. Three nodes in one rack is not fault tolerance; it is three copies of one failure.

The subtler failure

A leader that is alive but pathologically slow — long garbage-collection pause, disk saturation, network congestion. It keeps its lease by a hair, so no election happens, and it serves every request badly. Liveness detection based only on "is the process responding" misses this entirely.

Detection has to include progress, not just presence: is the commit index advancing, is the queue draining, is latency within bound. A leader failing a progress check should step down voluntarily.

The design consequence

Choose consensus for the small set of decisions that genuinely need it — cluster membership, shard ownership, configuration — and keep it off the data path. Consensus is expensive, and the correct amount of it in a system is the minimum that makes everything else safe. Systems that route all writes through a consensus group inherit its latency and its availability characteristics for operations that never needed either.