A marketplace seller edits a listing and is immediately shown the old version, because the read went to a replica. What are the options, and how should the choice be made?
Show the full answer Hide the answer
Why this happens
Reads are spread across asynchronous replicas for scale. The write commits on the primary and propagates with a lag — usually milliseconds, occasionally seconds, and during heavy write load or a network event, much longer. A read issued immediately after a write frequently arrives at a replica that has not yet seen it.
The user's model is simple and correct: I just changed this; it should be changed. Violating read-your-writes is the most jarring consistency failure in any product, because it looks like the system lost the work.
The options
1. Route the user to the primary after a write. For a short window — a few seconds — that user's reads go to the primary. Simple, effective, and the standard answer. Cost: some primary read load, proportional to write rate rather than read rate, which is exactly the traffic profile that makes it affordable.
2. Read-your-writes via replication position. The write returns a log position; subsequent reads require a replica at or beyond it, or fall through to the primary. Precise and gives the guarantee only where needed, at the cost of plumbing the token through the client and the routing layer.
3. Return the write's result and render from it. The edit response contains the new state, and the UI renders that rather than re-reading. Removes the read entirely, is essentially free, and works only for the immediate page — a navigation moments later hits a replica again.
4. Optimistic UI with reconciliation. Show the intended state immediately, reconcile when the read catches up. Excellent for perceived latency, and it lies if the write actually failed — requiring careful error handling to avoid showing state that does not exist.
How to choose
By whose write and whose read:
- A user reading their own write — must be consistent. Use option 1 or 2. This is not negotiable; it is what users perceive as basic correctness.
- A user reading someone else's write — eventual is fine. A buyer seeing a listing edit a second late is invisible.
- A system making a decision — depends entirely on the decision. Displaying a listing tolerates lag; validating at purchase does not, and must read the primary or use a conditional write.
The operational half
Whatever is chosen, replication lag must be monitored and acted on. Two rules that pay for themselves:
- Remove a lagging replica from the read pool past a threshold. A replica minutes behind is worse than no replica, because it serves confidently wrong data.
- Alert on lag, not just on replica health. A healthy, connected, badly lagging replica is the common failure and passes every liveness check.
The lesson
Replication lag is not a bug to eliminate; it is a property to design around. The architecture's job is to identify the small set of reads that genuinely require freshness and route only those to the primary, rather than either ignoring the problem or abandoning replicas entirely.