Replication

Definition

Replication copies data from a primary to one or more replicas. Synchronous replication waits for acknowledgement before committing; asynchronous does not. The choice determines whether you can lose committed data on failover, and how much latency each write costs.

What replication is actually for

Three distinct purposes, frequently conflated:

• Availability. A replica can be promoted when the primary fails.
• Read scaling. Reads distributed across replicas.
• Locality. A replica near the user reduces read latency.

They have different requirements. Availability needs the replica to be current enough that failover does not lose data. Read scaling needs enough replicas. Locality needs geographic placement, which maximises lag. A design that assumes one mechanism serves all three usually disappoints at the third.

Replication lag, and why it is a product problem

Asynchronous replicas are behind. Usually milliseconds; during a bulk load or a long transaction, seconds or minutes. The user-visible symptom is the read-after-write problem: a user updates their profile, the read goes to a replica, and their change is not there. They update it again.

Mitigations, in increasing order of cost:

• Route a user's reads to the primary for a short window after they write. Cheap, effective, and covers the overwhelming majority of complaints.
• Sticky routing to one replica per session, giving monotonic reads — the user never sees time go backwards, even if they see slightly old data.
• Carry the write position and wait for the replica to reach it. Precise, and adds latency.
• Read from the primary for anything that must be current, accepting the load.

Industry example

Meta's read path is built on the assumption that reads outnumber writes by orders of magnitude, so the architecture is layered: a small write-serving tier, a large fleet of read replicas, and an enormous distributed cache in front of both.

The instructive part is cache and replica invalidation as its own distributed problem. When a write commits, invalidations must reach caches in every region, and they race with reads that are already in flight. Solving that requires invalidation to be ordered with respect to the write — which is why cache invalidation is repeatedly described as one of the genuinely hard problems rather than as a cache configuration detail.

The transferable lesson: at high read fan-out, the replica is not the last layer, and each additional layer adds another window in which a stale value can be served.

Failure scenarios

• Failover with asynchronous replication loses committed transactions — the acknowledged writes that had not yet reached the replica. This must be an accepted, quantified RPO, not a surprise.
• Split brain. Both nodes believe they are primary and both accept writes. Requires fencing — the old primary must be actively prevented from writing, not merely asked to stop.
• A replica used for a heavy report falls behind, and then is promoted during an incident.
• Lag unmonitored, so failover happens onto a replica that is twenty minutes stale.

Trade-offs

Synchronous: no data loss, higher write latency, and reduced availability — an unreachable replica can block writes unless the quorum is configured to tolerate it. Asynchronous: fast writes, bounded data loss on failover. Most systems use synchronous replication to one nearby replica and asynchronous to distant ones, which is a reasonable default worth stating explicitly.

Interview question

"Users report that their own edits sometimes disappear after saving. Replication lag is 300 ms. Explain the mechanism and give me three fixes with different costs."