Users report their own profile edits sometimes vanish immediately after saving. Replication lag averages 300 ms. Explain and give three fixes of increasing cost.
Show the full answer Hide the answer
What is being tested
The read-after-write problem, and whether you reach for the cheap targeted fix before the expensive general one.
The mechanism
The write goes to the primary. The subsequent read is load-balanced to a replica that has not yet applied it. The user sees their old data, concludes the save failed, and saves again — often creating a second problem.
Note what makes this specifically bad: the user is the one person guaranteed to notice, because they know exactly what they just changed. Other users seeing 300 ms stale data notice nothing. So the fix does not need to make the whole system consistent; it needs to fix one user's view of their own writes.
Three fixes, cheapest first
1. Route a user's reads to the primary for a short window after they write. Set a marker — a cookie, a session flag, a short-lived cache entry — when a user performs a write, and route their reads to the primary for, say, ten seconds. Cheap, easy, and eliminates essentially all user complaints. Cost: a small increase in primary read load, proportional to write rate rather than to read rate, which is why it is affordable.
2. Sticky routing per session, giving monotonic reads. Pin a session to one replica. The user may see slightly old data, but they never see time go backwards — which is the property that makes staleness tolerable rather than alarming. Cost: uneven replica load and a rebalancing problem when a replica is removed.
3. Carry the write position and wait. The write returns a log position; subsequent reads pass it and the replica waits until it has applied at least that far. Precise and correct. Cost: added latency on reads, an application protocol to thread the position through, and engine support.
A fourth, sometimes right: read from the primary for the specific views where currency matters — account settings, order status — and from replicas for everything else. This is often the most honest answer, because it makes the consistency decision per screen rather than globally.
What a strong answer adds
Lag is not constant. It averages 300 ms and spikes to minutes during a bulk load, a long transaction, or a schema migration. Any fix based on a fixed delay will fail exactly when the system is under stress. Fixes 1 and 3 are robust to that; a "wait 500 ms then read the replica" hack is not.
Monitor lag and alert on it. Failing over to a replica that is twenty minutes behind, during an incident, is how a bad hour becomes a bad week.
At high read fan-out there are more layers than the replica. Caches in front of replicas add another staleness window, and invalidations race with reads already in flight — the problem Meta's caching work is well known for. Each layer added multiplies the ways a stale value can be served, so the read-your-own-writes fix has to apply at every layer, not just the database.