Where does consensus belong in an architecture, and where is it a mistake?
Show the full answer Hide the answer
What is being tested
Whether you place consensus where its cost is affordable.
What it buys and what it always costs
Buys: agreement on a value or an ordering among nodes despite failures — a linearizable foundation.
Costs: at least one round trip, and a majority quorum. Three nodes tolerate one failure; five tolerate two. Cross-region ensembles pay the round trip on every write.
And the corollary that catches people: two nodes can never form a quorum. A two-region deployment needs a witness in a third location, or it cannot decide anything during a partition.
Where it belongs
Small, low-volume, high-value state:
- Cluster membership and leader election.
- Configuration and feature state that must be globally consistent.
- Shard-to-node assignment.
- Distributed locks — with fencing tokens, because a lock holder that pauses for a long garbage collection may still believe it holds an expired lock.
The volume is low, the value of correctness is high, and the round trip is affordable because it happens rarely.
Where it is a mistake
High-volume application data. Routing every write through a single consensus group caps throughput at one ensemble's capacity and adds a round trip to every operation.
The usual architecture
Consensus for the control plane; partitioned and replicated stores for the data plane — with each partition's leadership decided by consensus, so the expensive mechanism runs once per leadership change rather than once per write.
That structure is why distributed databases can offer strong guarantees without paying consensus cost on every operation.
What a strong answer adds
That leader election is only half the problem: fencing is the other half. A deposed leader that is unreachable to the cluster may still be reachable to clients, so the storage layer must reject writes carrying a stale fencing token. Without that, leader election produces split brain rather than preventing it.