Leader Election in Practice

Definition

The process by which a set of nodes agrees that exactly one of them holds a particular role — accepting writes, running a scheduled job, coordinating a partition.

Why it matters

Many problems become dramatically simpler with a single coordinator: ordering, exclusive access, scheduling, and avoiding duplicate work. The difficulty is that "exactly one" must hold even during a network partition, when nodes cannot communicate to agree.

Getting it wrong produces split brain — the most damaging distributed failure, because unlike an outage it does not announce itself. Both halves appear healthy, both serve traffic, and the damage surfaces when the partition heals and two histories cannot be merged.

Implementation patterns

Majority quorum. A leader requires votes from more than half the nodes. A partition can produce at most one majority, so at most one leader. This is the foundation and everything else is detail.

Consensus protocols — Raft, Paxos and their variants — implementing this correctly including leader election, log replication and membership change.

Lease-based leadership, where the leader holds a time-bounded lease it must renew. Simple, and unsafe on its own because a paused process can believe it still holds an expired lease.

Fencing tokens. The lock service issues a monotonically increasing number with each grant; every write carries it, and the protected resource rejects any token lower than the highest it has seen. This is what makes leadership safe, because correctness moves to the resource rather than depending on the leader's belief about time.

Failure scenarios

Two-node clusters. Two nodes cannot form a majority — any rule one side satisfies, the other satisfies too. A two-region active-active deployment needs a third site as witness, and its absence is the most common structural flaw in two-region designs.

Process pause. A leader stalls for garbage collection or scheduler preemption, its lease expires, a new leader is elected, and the old one wakes up and writes. No component did anything wrong, and timeouts cannot fix it — only fencing can.

Failover thresholds shorter than transient network events, causing failover for every brief blip.

No fencing on the resource. Where the protected resource cannot check a token — a filesystem, a legacy API, a third-party service — leader election cannot be made safe, and the honest answer is to restructure so mutual exclusion is not required.

Operators overriding the minority. The minority side refusing writes is the design working as intended, and it is what gets overridden under pressure by someone who sees a healthy node declining traffic.

Industry example

Roblox's 2021 outage illustrates the other side of the trade: the Consul cluster provided consistent coordination for the entire platform, and when it degraded under a novel load pattern, everything depending on it failed for roughly 73 hours. Centralised coordination is valuable and makes the coordinator's availability an upper bound on the platform's.

Trade-offs

A leader simplifies enormously and introduces a single point of coordination, a failover window during which nothing progresses, and a component whose availability bounds the system's.

The alternative — designing so no leader is needed, through partitioned single-writer ownership or commutative operations — avoids the whole problem and is available more often than teams assume.

Interview question

A team designs active-active across two regions with automatic failover based on health checks. What is wrong?

The candidate must identify that two nodes cannot form a majority, so a partition is indistinguishable from failure and both sides can promote. Strong answers propose a third-site witness, single-writer regional partitioning as the frequently-better alternative, and fencing so a deposed primary's writes are rejected at the storage layer.