A 3-node Raft cluster in a single data centre must become 5 nodes across three data centres, live, with no unavailable window. Give the sequence, and say what goes wrong if the two new members are added in one step.

Why one step is unsafe

A configuration change is not a configuration file edit, because the members do not switch configurations at the same instant. During the switch, some members believe the group is {A,B,C} and others believe it is {A,B,C,D,E}. A majority of the old configuration is 2; a majority of the new one is 3. {A,B} and {C,D,E} are both majorities of some configuration the cluster currently believes in, so two leaders can be elected and both can commit. That is split-brain produced by an administrative action rather than by a network fault.

Raft's two documented answers are joint consensus - a transitional configuration in which a decision needs a majority of both the old and the new membership, so no two disjoint majorities exist - and the simpler single-server change, where adding or removing one member at a time makes overlap automatic. The dissertation (2014) recommends the single-server form for exactly this reason, and most implementations expose only that.

The sequence

1. Add D as a non-voting learner. It receives the log and counts for nothing. No quorum changes, so there is no risk window.
2. Wait for its lag to fall below one heartbeat interval, verified from the leader's view of its match index, not from the node's own claim of health.
3. Promote D to voter. Quorum goes from 2 to 3. Reversible: demote it.
4. Add E as a learner - catch up - promote. Quorum stays 3, tolerance goes from 1 to 2.
5. Only now move members between data centres, one at a time, each as add-learner / promote / demote-and-remove the old one.

Each step is a single-member change, so the old and new majorities always share a member, and each step is reversible until the next one starts.

What changes the moment the group spans sites

The commit now waits for a cross-site acknowledgement, because any 3-member majority of a 2-2-1 or 2-1-2 layout includes at least one remote member. Commit latency goes from a same-rack fsync, under a millisecond, to the round trip to the second site - 2 to 30 ms between metropolitan sites, and 60 to 100 ms between continents. Teams are surprised by this and should not be: it is the cost of the durability they asked for.

Put the leader where the writes are, and prefer three sites that are close over two that are close and one that is far: the deciding acknowledgement is the median member, so one very distant voter is nearly free while two make every commit slow.

Where it can diverge, and how you would know

Nothing diverges if each step is a single-member change - the protocol guarantees it. What does go wrong operationally: a learner promoted while still far behind becomes a member that cannot serve as part of a quorum, so a second failure is now fatal when the dashboard says the cluster is 5-node healthy. Watch match-index lag per member and raw member count, and alert on voters-in-sync rather than on members-configured.

The point of no return

Removing the last member in the original data centre. Before that, you can demote the new ones and go back; after it, the original site holds no voter and rollback means a fresh membership change under whatever conditions obtain then.

When this is the wrong answer

If a 30-second write pause is acceptable - and for a metadata store that nothing user-facing blocks on, it often is - stop the cluster, change the configuration, start it, and verify. It is one step instead of eight, it has no half-states, and the failure mode is a clean restart rather than a partially-promoted member nobody can reason about at 2 a.m.