Witness Replica

A company runs two data centres 20 ms apart and wants automatic database failover with no data loss. With one replica in each site there is no safe answer: either site can lose contact with the other and cannot tell "the peer is down" from "I am the one that is cut off", so promoting automatically risks two primaries and not promoting means no automatic failover at all. A third full replica would settle it, and a third data centre with a third copy of a large database is expensive and sometimes not available.

A witness breaks the tie without carrying the data. It participates in elections and in commit quorums for metadata purposes, holds no user data or only the log, and runs happily on a small instance in a third location - another availability zone, a small colocation, a cloud region used for nothing else.

Why it matters

Safety in a quorum system comes from voters in an odd number of failure domains. Durability comes from copies. Teams conflate the two budgets, which produces either a two-site design that cannot fail over safely, or a three-site design that pays three times for storage to buy one vote.

Separating them is what makes two-site automatic failover possible at all, and it is why the pattern appears in nearly every mature replicated store: a cheap vote in a third place is the difference between a documented failover procedure and a human decision taken under pressure at 3 a.m.

Implementation patterns

• Vote-only member, holding the replication log but no state machine. It can vote and it can never be promoted, which is exactly the intent.
• Witness in a third failure domain, always. A witness in one of the two data centres buys nothing: losing that site loses both a data copy and the tiebreaker at once.
• Metadata-only witness, as some distributed databases implement, where the witness stores per-partition commit metadata so it can participate in commit quorums without holding rows. Cloud Spanner's witness configurations work on this principle.
• Explicit promotion policy. Write down - and automate - what happens when one data replica is lost: the surviving pair of (data replica + witness) is a majority and may continue, and the fact that there is now one copy must raise an alarm, not pass silently.
• Size it for the log, not the data, and monitor it like a voter: a witness that is down turns the cluster back into a two-member group with no safe failover, and nobody notices because the data path is unaffected.

Industry example

MongoDB's arbiter is the most documented instance, and the documentation is equally explicit about its risk: with a primary, one secondary and an arbiter, a majority write concern can be acknowledged while only two copies exist, and losing the secondary leaves the data in one place while the set still believes it has a majority. MongoDB's own guidance is therefore to prefer a third data-bearing member and to use arbiters only where the cost argument is decisive.

Cloud Spanner's regional configurations use witnesses to achieve an odd number of voting replicas across an even number of read-write locations. Both cases teach the same lesson in opposite directions: a witness makes failover safe and leaves durability exactly where it was.

Failure scenarios

• The silent single copy. One data replica fails, writes continue on the survivor plus witness, and the deployment is one failure from data loss while every dashboard reads healthy.
• Witness in the wrong place, inside one of the two main sites, so a site loss takes the tiebreaker with it.
• Witness unmonitored, down for weeks, discovered during the failover it existed to enable.
• A witness promoted by a tool that does not know it holds no data, producing an empty primary - which is why vote-only members must be ineligible by configuration, not by convention.
• Cross-site latency assumed irrelevant because the witness holds no data: it still has to acknowledge, so a witness 200 ms away can become the deciding vote when one local replica is slow.

Trade-offs

The honest framing: a witness converts an availability problem into a durability compromise that you accept explicitly. That is a reasonable trade when the data is also backed up and replicated asynchronously elsewhere, and a poor one when the witness is the whole plan.

When not to use it

When a third site with real capacity is available, use a third data replica - it costs more and removes the single-copy window entirely. When write concern genuinely requires two durable copies at all times, a witness cannot provide it, and configuring one anyway produces a system that acknowledges writes it has not made durable twice.

And when there is no automatic failover requirement at all, the witness is unnecessary machinery: two copies and a documented manual promotion is simpler and has no hidden degradation state.

Interview question

Q: You have two data centres and a business requirement for automatic database failover with RPO near zero. Design the topology, then tell me what your monitoring must alert on that a three-replica deployment would not need.

What a strong answer covers: two data replicas plus a vote-only member in a third failure domain; why the witness must not live in either main site; the post-failure state in which only one copy exists and why that needs its own alert rather than a log line; witness liveness as a first-class signal; the acknowledgement path, so that a distant witness does not become the deciding vote; and the explicit statement that this buys availability and not durability.

Quick check

Quiz: Why must a witness replica never be placed in one of the two primary data centres? Because losing that site would remove both a data copy and the tiebreaking vote at once, leaving the survivor in a minority and unable to fail over.

Flashcard: What does a witness buy and what does it explicitly not buy? It buys an odd voter count for safe automatic failover across two sites; it buys no additional copy of the data, so redundancy after one failure is one.