A team runs a 3-node etcd cluster and adds a fourth node so it can "survive more failures". Write latency rises slightly and the cluster still tolerates only one node loss. Which statement explains what they did?
Show the full answer Hide the answer
What is being tested
Whether you can do the quorum arithmetic without looking it up. Cluster sizing is the one consensus decision almost every team makes, and the "more nodes is safer" instinct is wrong in a specific, arithmetical way.
The mechanism
A consensus group commits an entry once a strict majority of its voting members have
persisted it. Majority means floor(n/2) + 1, and fault tolerance is what is left over:
n - quorum.
| Members | Quorum | Losses tolerated |
|---|---|---|
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
| 6 | 4 | 2 |
| 7 | 4 | 3 |
Tolerance only improves on odd steps. Every even member is a node you pay for, replicate to and wait behind, in return for nothing. It also makes a tie possible in an election, which costs an extra election timeout rather than correctness, because Raft breaks ties by randomised timeouts and tries again.
The latency effect is the second half of the answer. A commit waits for the quorum, so it is governed by the slowest member of the fastest majority - effectively the median acknowledgement. Going from 3 to 4 moves the deciding acknowledgement from the fastest of two followers to the second-fastest of three, so p99 writes get slightly worse.
Why the other options fail
- "Quorum is half the members." Half of four is two, and two disjoint pairs could both believe they had agreement: that is split-brain. The strictness of the majority is what guarantees any two quorums share at least one member, which is the entire safety argument.
- "Even-sized clusters cannot elect a leader." They can. A split vote wastes one randomised election timeout, typically 150 to 300 ms in a Raft implementation, and then one candidate wins. Even sizes are wasteful, not broken.
- "The leader waits for every follower." If it did, availability would be worse than a single node, because any one failure would stop commits. Waiting for a majority is precisely how the protocol survives a minority being down or slow.
When this is not the rule to apply
A cluster of two data replicas plus one vote-only member in a third location is deliberately chosen, and it reads like an even design because only two members hold data. It gives a 3-member quorum without paying for a third full copy. The rule is about voters, not about data: keep the number of voters odd and put them in an odd number of failure domains.
What a strong answer adds
Say what you would measure before resizing: the commit-latency distribution per follower, and whether the previous failure was really a node loss rather than a correlated one. Three nodes in three racks of one data centre survive a node; they do not survive the data centre. Adding a fourth node in the same room buys nothing against the failure that is actually coming.