Replication in Practice

Definition

Maintaining copies of data on multiple nodes. The design space is defined by two questions: who may accept writes, and when is a write acknowledged relative to its propagation.

Why it matters

Replication serves three distinct purposes that are frequently conflated:

Durability — surviving the loss of a node without losing data. Availability — continuing to serve when a node fails. Read capacity — spreading read load across copies.

They pull in different directions. Read capacity wants many replicas serving reads, which increases staleness. Durability wants writes acknowledged only after they are safely on several nodes, which increases write latency.

Implementation patterns

Single primary with asynchronous replicas. Writes go to one node and propagate in the background. Fast writes, scalable reads, and a failover can lose recent writes. The most common configuration.

Single primary with synchronous replication to at least one replica. No data loss on failover, write latency includes the round trip.

Quorum-based, where a write succeeds once acknowledged by W of N replicas and a read consults R, with W + R > N guaranteeing overlap. Tunable per operation.

Multi-primary, where several nodes accept writes. Maximum availability and write locality, and it requires conflict resolution because two nodes can accept conflicting writes concurrently.

Failure scenarios

Read-your-writes violated. A user updates their profile, the read is load-balanced to a lagging replica, and the old value appears. They update again — and now there is a support ticket and a belief that the system loses data. This is the most common user-visible replication bug and it is entirely preventable with session guarantees: route reads to the primary briefly after a write, use sticky routing, or have the write return a version token the client presents on subsequent reads.

Silent replica lag. Lag grows without alerting, and reads become arbitrarily stale. Lag must be monitored in time, not in bytes or transactions.

Failover with unbounded loss. Asynchronous replication means failover discards whatever had not propagated. That quantity should be a known, monitored number, not a discovery.

Split brain where a partition allows two nodes to believe they are primary. Requires quorum and fencing.

Industry example

GitHub's 2018 incident is the canonical warning: a 43-second network partition triggered automated cross-region promotion, both sides briefly accepted writes, and reconciling the divergence took over 24 hours. The lesson is that failover thresholds must be set against the duration of transient network events, and that a deposed primary must be positively fenced.

Trade-offs

More replicas give better read capacity and availability, and worse write latency under synchronous replication, more staleness under asynchronous, and more cost.

Synchronous protects against data loss and couples write availability to replica availability. Asynchronous is fast and accepts bounded loss on failover.

Interview question

After a user updates their profile the page sometimes shows the old value. The team says this is normal eventual consistency. Is that acceptable?

The expected answer is no — read-your-writes is a session guarantee separable from the global consistency model, so this is a fixable defect rather than an inherent cost. Strong candidates enumerate the fixes and note that monotonic reads and consistent prefix are the related guarantees worth providing.