Joint Consensus

A team doubles a 3-node coordination cluster to 5 by editing the member list and restarting the new nodes. For a few hundred milliseconds, some members believe the group is {A,B,C} and others believe it is {A,B,C,D,E}. A majority of the old group is two members; a majority of the new one is three. {A,B} and {C,D,E} are each a majority of a configuration the cluster currently believes in, so both can elect a leader and both can commit - and the two histories diverge with no protocol-level way to reconcile them.

Nothing failed. No network partitioned. The split was produced by an administrative action, and it is the reason membership change is treated as a protocol problem rather than a configuration problem.

Joint consensus is the fix: the cluster first commits a special entry describing the union of the two configurations. The dangerous window is often under 200 ms in a healthy cluster, and that is long enough, because one uncommitted entry on each side is all divergence takes. While that entry is in force, a decision requires a majority of the old membership and a majority of the new one, which no two disjoint sets can satisfy. Once the joint entry is committed, the cluster commits a second entry for the new configuration alone.

Why it matters

Every consensus deployment is reconfigured eventually: hardware is replaced, a site is added, a node is retired. The dangerous window is not the new node joining, it is the interval in which two definitions of "majority" coexist, and that interval exists whether or not the operator knows about it.

The failure is also the worst kind to detect. Two leaders committing to the same group produce divergent logs that both look internally consistent. There is no error, no alert and no automatic repair, and the discovery usually comes from a downstream reconciliation days later. Protocols that make this impossible by construction remove a whole class of operator error.

Implementation patterns

• Joint consensus proper, as in the original Raft presentation: commit C_old,new, then commit C_new. It supports arbitrary changes - 3 members to 5 in one operation - at the cost of a transitional state the implementation must handle in elections, log replication and leadership transfer.
• Single-member change, which the Raft dissertation (2014) recommends instead: add or remove exactly one member per operation. Majorities of consecutive configurations always overlap in at least one member, so no transitional state is needed. Most production implementations expose only this.
• Learners first. Add the member as a non-voting learner, let it replicate the log, and promote it only when its match index is within a heartbeat of the leader. Promotion is then a formality rather than a gamble, and the learner phase costs no quorum risk at all.
• Leadership transfer before removal. Removing the current leader works, and it costs an election; transferring leadership first turns an outage of one election timeout into a controlled handover.
• Reject concurrent changes. A correct implementation refuses a second membership change while one is uncommitted. If yours does not, that discipline has to live in the runbook.

Industry example

etcd exposes single-member changes and a learner role, and its documentation is explicit that a member must be added and removed one at a time; the learner exists precisely so that a slow new member cannot be counted in a quorum before it has caught up. The operational guidance that matters is the quorum arithmetic during the change: a 3-member cluster being grown to 5 passes through a 4-member state whose quorum is 3, so during that step the cluster tolerates one failure rather than two, and a second failure at that moment stops writes.

Kubernetes clusters inherit this directly, because etcd is their store: a control-plane upgrade that replaces members one at a time is following this rule.

Failure scenarios

• Two-at-once growth, producing disjoint majorities and two committed histories.
• Promoting a learner that is still far behind, giving a cluster that reports five healthy members but has only three that could serve a quorum.
• Removing the last member in a site before verifying that the remaining voters span an odd number of failure domains, so the next site failure is a quorum loss.
• A membership change attempted during a partition, where the new configuration commits on one side only and the other side continues under the old one.
• Configuration drift, where the committed membership and the operator's inventory disagree, so a "safe" removal takes out a voter nobody knew was load-bearing.

Trade-offs

The deeper trade is availability against operator simplicity. Live reconfiguration keeps the service up and asks the operator to understand an eight-step sequence; offline reconfiguration asks for a maintenance window and is nearly impossible to get wrong.

When not to use it

If a short write pause is acceptable - which it often is for a metadata or coordination store that no user request blocks on - stop the cluster, change the configuration, restart and verify. One step with no intermediate states beats eight steps whose failure modes are difficult to diagnose at 2 a.m.

And if the group is managed by a platform that performs rolling member replacement for you, the right action is to understand what it does rather than to drive it by hand. Hand-running membership changes against a managed consensus service is how two uncommitted changes end up in flight at once.

Interview question

Q: You need to move a 3-node etcd cluster from one data centre to another with no write downtime. Walk me through the sequence, say what the cluster tolerates at each step, and tell me where you would stop and roll back.

What a strong answer covers: add-learner, catch up, promote, remove - one member at a time, never two; quorum arithmetic at each step, including the 4-member state whose tolerance drops to one; verification by match-index lag rather than by node self-report; leadership transfer before removing the leader; the point of no return as the removal of the last original-site voter; and the observation that commit latency changes the moment a majority must cross sites.

Quick check

Quiz: Why can changing a Raft group from 3 members to 5 in a single step elect two leaders? Because a majority of the old configuration (2) and a majority of the new one (3) can be disjoint sets, and both are valid majorities while the two configurations coexist.

Flashcard: What does the joint configuration require of every decision? A majority of the old membership and a majority of the new one simultaneously - which no two disjoint sets can satisfy.