Pre-Vote

A cluster shows leader changes several times an hour. Writes return "leader changed" in small bursts, read leases are invalidated, client connections reset. No node is failing, the network looks healthy in aggregate, and the elections correlate with nothing anyone can find.

The cause is usually a member with a flapping link. While isolated it campaigns, fails, and campaigns again, incrementing its term each time - and in Raft the term number outranks everything. When the link heals, it broadcasts a vote request at a term hundreds higher than the cluster's. Every member, the leader included, follows the rule "on seeing a higher term, step down and adopt it". The returning node then loses the election, because voters reject a candidate whose log is behind, and the cluster elects a leader again at the new term - usually the one it just deposed.

Pre-vote removes the mechanism. Before incrementing its term, a candidate asks the cluster a hypothetical: would you vote for me if I asked? Voters answer using the rules they would really apply, including the log-up-to-date check. Only if it could win does the candidate increment its term and campaign for real, so a node with a stale log never raises the term and never disturbs anything.

Why it matters

The individual cost is small - one election timeout of write unavailability, typically 150 to 300 ms in a Raft implementation - and that is precisely why the problem persists. No single occurrence justifies an investigation; the aggregate is what hurts.

Each leadership change invalidates read leases, so a system using lease reads loses its fast read path for an election timeout. In-flight batches abort and are retried. Clients that pin connections to the leader reconnect. A cluster that changes leaders twenty times an hour is not broken in any way a health check can express, and it is measurably worse at its job.

Implementation patterns

• Pre-vote on by default. etcd enables it, and a Raft implementation that does not offer it has a known availability defect.
• Pair it with check-quorum, which covers the opposite direction: a leader that has not heard from a majority within an election timeout steps down on its own, instead of continuing to claim leadership and serve leases from the minority side of a partition.
• Randomised election timeouts remain necessary; pre-vote prevents disruption by a node that cannot win, not split votes between nodes that can.
• Graph leader-change rate per hour and alert on a step change, never on a single election. The single election is normal; the rate is the signal.
• Leadership transfer for planned events. Draining a node for maintenance should hand leadership over deliberately rather than let the election machinery discover the loss.

Industry example

The mechanism is described in Diego Ongaro's Raft dissertation (2014), where the disruptive-server problem is set out explicitly: a server that has been removed from the cluster, or that is returning from isolation, can disrupt an otherwise healthy cluster purely through its term number. etcd's implementation carries both pre-vote and check-quorum, and its documentation treats them as the pair that makes leadership stable across flapping networks.

The lesson generalises beyond Raft. Any protocol in which a monotonically increasing number grants authority has this hazard, and the fix always has the same shape: check eligibility before claiming the authority, rather than claiming it and being rejected afterwards.

Failure scenarios

• Flapping link on one member, producing persistent leadership churn with no correlated cause - the canonical symptom.
• A removed member that never got the memo, still campaigning at ever-higher terms against a cluster it is no longer part of.
• Slow disk mistaken for isolation. A member that misses heartbeats because its fsync is slow campaigns for the same reason an isolated one does, and here pre-vote suppresses a symptom that was telling you something true.
• Check-quorum missing, so an isolated leader keeps serving lease reads from stale state - the more dangerous half of the pair, because it is a correctness issue rather than an availability one.
• Alerting on individual elections, which trains the team to ignore the alert long before the rate becomes a problem.

Trade-offs

Pre-vote adds one extra round of messages to every genuine election, so real failover is marginally slower - one additional round trip - in exchange for not having spurious failovers at all. For any cluster where leadership changes are rarer than link flaps, which is nearly all of them, that trade is obviously right.

The subtler cost is diagnostic. Pre-vote makes a class of misbehaving member invisible: the node with a slow disk or a flapping NIC no longer announces itself through leadership churn, so it keeps sitting in the quorum and setting commit latency. The compensating control is per-member commit-latency and heartbeat-miss metrics, which the churn used to provide for free.

When not to use it

There is no deployment where disabling pre-vote is the right default. The honest "when not" is about expectations rather than configuration: do not treat pre-vote as a fix for leadership instability without first checking why a member is campaigning. If the answer is a degraded disk or a saturated network interface, the node needs replacing, and pre-vote has only hidden the evidence.

Equally, pre-vote does nothing for split votes between healthy candidates, for slow elections caused by large snapshots, or for client-side retry storms after a legitimate failover. Those need randomised timeouts, snapshot tuning and client backoff respectively.

Interview question

Q: A five-node cluster changes leaders fifteen times an hour. Every node reports healthy, no node is restarting, and the elections do not line up with deploys or traffic. How do you investigate, and what would you change?

What a strong answer covers: the term-number mechanism by which a node that cannot win still forces a step-down; checking per-member heartbeat misses and commit latency to find the flapping or slow member; pre-vote and check-quorum as the protocol-level fixes and which direction each one covers; the costs that make churn matter even though each election is milliseconds - lease invalidation, aborted batches, client reconnects; and the warning that pre-vote can mask a genuinely degraded member, so the per-member metrics must stay.

Quick check

Quiz: Why does a node that is certain to lose an election still cause an outage without pre-vote? Because it increments its term before asking for votes, and every member - including the healthy leader - steps down on seeing a higher term, so the cluster must hold an election it did not need.

Flashcard: What question does a pre-vote ask, and what does the candidate do with the answer? "Would you vote for me if I asked?" - and only if a majority would, does it increment its term and start a real election.