Your write hasn't happened here yet
How production systems give a client back its own write when reads are served by replicas that trail the primary, and how those mechanisms fail.
Reconstructs the four-part machine behind read-your-writes (mint a position token at commit, carry it with the client, route to a caught-up replica, keep a bounded escape hatch) from the implementations at GitLab, Wikimedia, Rails, MongoDB, MaxScale, Cloudflare D1 and LiteFS, and from four postmortems that show the failure classes. A reader leaves able to choose time-based versus position-based sticking deliberately, decide what the timeout does before it happens, and instrument the two failure modes that raise no alarm.
Every layer of the stack independently converged on the same four-part token machine, while the database underneath took three landing attempts across six visible years of commits to ship the one-sentence primitive below it.
What you get out of it
- A stale read raises no alarm: GitLab ran a severity-2 incident for 37 hours because the only symptom was CI pipelines stuck in 'running', and found it via a routing-invariant metric, not an error rate.
- Fallback-to-primary is safe only while lag is uncorrelated with load; when overload causes the lag, the fallback aims the whole read fleet at the bottleneck (two GitLab severity-1s in October 2025).
- Treat unknown lag as infinite: Vitess coerced a NULL lag signal to zero and unboundedly stale replicas passed health checks.
- For asynchronous work the token must travel with the job, not the user session: GitLab enqueues jobs with the primary's LSN and defers or promotes on replica miss.
- Every token dies young by design (Rails 2 s, MediaWiki ~10 s, GitLab 30 s, LiteFS 5 min); an immortal token silently converts the replica tier into decoration.
Scope
Why this, now. PostgreSQL 19's WAIT FOR command finally lands the database-level primitive in 2026 after two reverted attempts, and it was still receiving recovery-deadlock fixes twelve days before this guide was researched.
What it does not cover. Multi-primary writes and conflict resolution, cache-tier staleness (covered by the cache-miss dig in this series), geo-partitioned data placement, and the paper/talk/blog evidence layers, whose hosts were unreachable under this session's network policy.
Other field guides
How long to wait: timeouts and deadlines in distributed systems
A field guide to timeouts and deadlines: the difference between a local per-call timeout and a propagated deadline, the stack of independent timers e…
22 sources · 13 organisations · 3 postmortemsWhen two hold the lock: what the repositories admit about distributed mutual exclusion
Every mainstream distributed lock ships with a written admission that it cannot guarantee mutual exclusion, and this guide reads those admissions whe…
22 sources · 13 organisations · 3 postmortemsWhen the answer is wrong, not late: ten years of CockroachDB read from its own record
A decade of one distributed database company read from artefacts nobody edits for effect: technical advisories with affected version ranges, design r…
26 sources · 3 organisations · 8 postmortems