How production systems give a client back its own write when reads are served by replicas that trail the primary. Reconstructed from GitLab's public incident record, the source and issue trackers of Rails, MediaWiki, Vitess, ProxySQL, MaxScale, MongoDB, CockroachDB, Cloudflare D1 and LiteFS, and ten years of PostgreSQL commits. After reading, you can pick a read-your-writes mechanism deliberately, size its escape hatch, and know which two failure modes to instrument for, because neither raises an alarm on its own.
A system accepts a change, confirms it, and then answers the very next question from a copy that has not yet heard about the change. Every architecture that scales reads by replicating data owns this problem; this page maps how real systems answer it, and how the answers fail.
runningStart with the incident, because it shows the stakes precisely. On 7 October 2024 at
20:50 UTC, GitLab.com pipelines began sticking permanently in the running
state. Nothing errored. No availability or latency graph moved in a way that pointed
anywhere. The incident
review reports that a refactor had caused Sidekiq middleware to route "certain queries
to a replica db that should have been sent to the primary db, causing those queries to see
slightly stale data (missing very recently committed transactions)". A worker asked a
replica whether a pipeline needed processing a few milliseconds after the primary recorded
that it did. The replica said no. The worker exited, no retry was scheduled, and the
pipeline stayed "running" forever. It took 37 hours and several premature
"mitigated" declarations to find, because a stale read is invisible to every standard
signal: the query succeeded, quickly, with well-formed rows.
That is the shape of the whole territory. Replication lag is usually well under a second (AWS states DynamoDB replicas converge "usually within one second or less"; GitLab's health thresholds are 8 MB or 60 seconds of lag before a replica is even pulled from the pool). The engineering effort documented on this page exists for the tail beyond that, and, more importantly, for the small class of reads where staleness does not degrade a page view but wedges a state machine: workflow engines, saga steps, webhooks, anything that decides not to act because it cannot yet see that it should.
Every layer of the stack has independently converged on the same four-part machine:
mint a position token at commit, carry it with the client, route reads to a copy that has
caught up to the token, and keep a bounded escape hatch for when none has. Wikimedia's
ChronologyProtector, GitLab's Redis sticking, MongoDB sessions, Cloudflare D1 bookmarks
and Fly.io's LiteFS cookie are the same design with different spellings. Meanwhile the
database that most of them sit on took three landing attempts across six visible years of
commits to ship the primitive underneath (WAIT FOR, PostgreSQL 19), and it
was still receiving recovery-deadlock fixes twelve days before this page was researched.
The concept is one sentence; the mechanism is genuinely hard at every layer.
Scope. This guide covers the read path of single-writer, asynchronously replicated systems: one primary, replicas that trail it, and the question of who may read where. It deliberately does not cover multi-primary writes and conflict resolution, cache-tier staleness and invalidation (a different dig in this series covers cache miss protocols), or geo-partitioned data placement. The vocabulary is Terry et al.'s 1994 "session guarantees" paper, which named read-your-writes; that paper and the later measurement literature were not reachable under this session's network policy, so everything below is built from incident reviews, source code, RFCs and issue threads that were fetched and are linked.
Across every implementation in the evidence, the mechanism decomposes into the same four parts. What varies is the spelling of the token, where it is carried, and what happens when the wait runs out.
Every serious implementation records where the primary was when the write
committed: a PostgreSQL LSN (GitLab stores the raw 0/16B3A78 form and
compares it in a Redis Lua script), a MySQL GTID set (MaxScale, ProxySQL, the Vitess
RFC), MongoDB's operationTime, D1's opaque bookmark, LiteFS's transaction
ID. The degenerate token is a clock reading: Rails stores
session[:last_write] and compares it to a two-second window. A timestamp is
cheap because the database does not have to participate; it is also a bet, not a
guarantee, and section 04 shows what the bet costs when it loses.
MediaWiki puts the token reference in a cookie
(cpPosIndex=<posIndex>@<write time>#<clientId>) so it
survives changing IPs and data centres; the position itself sits in a dc-local store
that must answer "within the order of one millisecond". GitLab keys Redis by user.
MongoDB and D1 put it in a driver session object and make the application carry it
between requests (D1's example ships it in an x-d1-bookmark HTTP header).
The most instructive variant is GitLab's job queue: the LSN is written into the job
payload, because for asynchronous work the reader is not the user's next request
but a worker seconds later on another machine. The carrier must follow whatever carries
the causality.
Routing is only as good as the pool it routes into. GitLab drops a replica from the candidate set at 8 MB or 60 seconds of measured lag (defaults; production tightens the latter to 30). Vitess vtgates "avoid serving data from instances that are lagging beyond X seconds". GitHub built a separate raft-replicated service, freno, whose lag measurements throttle writers so replicas stay valid for readers; its README states plainly that it exists for "mitigating write-then-read pains of master reads". The router then answers one question per read: is there a replica at or past the token? If yes, use it; GitLab "reverts to using replicas as soon as they have caught up, rather than always waiting the full 30 seconds".
When no replica has caught up, someone waits: MASTER_GTID_WAIT under
MaxScale (budget: causal_reads_timeout, default 10 s),
WAIT_FOR_EXECUTED_GTID_SET in the Vitess design, a 1 ms polling loop with
a 5 s ceiling in LiteFS's proxy, PostgreSQL 19's WAIT FOR on the standby
itself. The interesting design content is entirely in what happens at timeout. MaxScale
retries on the primary. GitLab's Sidekiq defers: the job is retried later, and only
promoted to the primary on the second miss. The Vitess RFC argues for failing the query
instead, "preventing cascading load during system stress". All three answers are
defensible; section 03 gives the condition that picks between them.
Two implementation details deserve attention because they mark the difference between a design sketch and a production system. First, expiry: every token dies young. Rails, two seconds. LiteFS's cookie, five minutes. GitLab's Redis keys, 30 seconds. MediaWiki's cookie, "about ten seconds", sized explicitly to cover "the cross-dc latency for those further away in any secondary DCs". An immortal token would pin every past writer to the primary forever and quietly convert the read-scaling tier into decoration; the fast modes of MaxScale carry exactly that warning, that under sustained writes "the traffic will end up being routed almost exclusively to the primary server". Second, the sticking store is itself a replicated data problem one level down. MediaWiki's doc block spells out the requirements: local reads must see local writes immediately, loss is survivable but visible, and the failure mode is honest: "users may be temporary confused as they observe their own actions as not being immediately reflected." GitLab compares LSNs inside an atomic Lua script so two racing requests cannot regress the stored position. When the 2024 incident broke this layer, nothing crashed; the machine kept running with the token pipeline severed.
Four forks recur across the record. Each block gives the observed choices and the condition that flips the answer.
local, MongoDB causal
sessions, D1 sessions, ChronologyProtector's per-client positions.universal fetches @@gtid_binlog_pos from the primary
before every read, so "the latency of any given SELECT statement increases by
roughly twice the network latency". etcd is the deliberate exception: linearizable
by default, because a coordination store's whole product is agreement.The fourth fork is where the primitive should live, and the record here is the surprise
this dig leads with. MySQL has shipped a session-scoped wait since 5.7
(WAIT_FOR_EXECUTED_GTID_SET, which every proxy above builds on). PostgreSQL's
commit history tells a harder story: an implementation of "waiting for given lsn at
transaction start" was reverted in April 2020; pg_wal_replay_wait() was
committed on 2 April 2024 for version 17 and fully reverted nine days later "per review by
Heikki Linnakangas" (version 17 shipped without the file); the WAIT FOR
command finally landed on 5 November 2025 for version 19, and the fix "Prevent WAIT FOR
LSN from deadlocking recovery on held locks" is dated 12 September 2026. The committer's
thread dates the effort to 2016, which this session could not verify directly; the six
years of visible history are enough to make the point. A wait primitive interacts with
snapshots, locks held across the wait, and recovery itself. If it takes the database this
long to get right, an application-layer version that polls and compares positions
(GitLab's, MediaWiki's, LiteFS's) is not a hack to be embarrassed about; it is the
pattern that shipped, a decade ahead.
| Decision | Common choice | Alternative | Because | Evidence |
|---|---|---|---|---|
| Token kind | Replication position (LSN/GTID/opTime/bookmark) | Wall-clock window (2 s / 5 s) | Positions are exact; clocks are cheap and need no DB cooperation | GitLab sticking.rb, Rails PR #35073 |
| Token carrier | Cookie or driver session | Server-side store keyed by user | Cookies survive DC and IP changes; stores need ~1 ms answers on every request | ChronologyProtector, MongoDB spec |
| Async work | Token travels in the job payload | Jobs always read the primary | Replica is "guaranteed to be caught up to the point at which the job was enqueued", else retry or primary | GitLab worker attributes |
| Timeout behaviour | Retry on primary (bounded) | Fail the read, or defer the work | Fallback is safe only while lag is uncorrelated with load | MaxScale, Vitess RFC, GitLab IR Oct 2025 |
| Default direction | Weak by default, strong opt-in (DynamoDB ConsistentRead, 2× cost) | Strong by default, weak opt-out (etcd; CockroachDB staleness bounds) | Follows what the product sells: throughput versus agreement | DynamoDB guide, etcd guarantees, CRDB RFC |
| Per-callsite choice | Forced and linted (every GitLab worker declares data_consistency) | A global default others silently inherit | Silent inheritance is how the wrong reads end up on replicas | GitLab worker attributes |
Four published incidents, four distinct failure classes: the routing rule breaks silently, the escape hatch amplifies, the health signal lies, and the lag itself pulls an operator into danger. None of the four announced itself as a consistency problem.
data_consistency: always) reach the primary.PipelineProcessWorker saw pre-write state.seconds_behind_master as NULL; Vitess coerced NULL to 0, so a replica that could not replicate at all looked perfectly caught up and kept serving.-unhealthy_threshold staleness bound was silently ineffective.The constants real systems chose, and what the incidents measured. Every row is linked; the provenance note below separates measured from claimed.
| Metric | Value | System | Context | As of | Source |
|---|---|---|---|---|---|
| Post-write primary window | 2 s | Rails default | Per browser session, after any write | 2019- | resolver.rb |
| Post-write primary window | 5 s + breaker | GitHub (production) | Plus a lag-checking circuit breaker that releases early | 2019 | Rails PR #35073 |
| Sticky-read expiry | 30 s | GitLab | Redis key TTL; released earlier once replicas catch up | 2026 | sticking.rb |
| Replica eviction thresholds | 8 MB / 60 s | GitLab defaults | max_replication_difference / max_replication_lag_time | 2026 | admin doc |
| Client token window | ~10 s | MediaWiki | Cookie lifetime sized to cover cross-DC replication | 2026 | ChronologyProtector |
| Position-store latency budget | ~1 ms | MediaWiki | The sticking store is on the path of nearly every request | 2026 | ChronologyProtector |
| Causal-read wait budget | 10 s default | MaxScale | causal_reads_timeout, then retry on primary | 2026 | ReadWriteSplit doc |
| Cluster-scope read tax | ~2× network RTT | MaxScale universal | Primary round-trip before every SELECT | 2026 | ReadWriteSplit doc |
| Replica poll / wait / cookie | 1 ms / 5 s / 5 min | LiteFS proxy | TXID polling interval, timeout, cookie expiry | 2026 | proxy_server.go |
| Strong-read price | 2× RCU | DynamoDB | 1 vs 0.5 read units per 4 KB; not offered on GSIs | 2023 | capacity doc |
| Typical replication convergence | ≤ ~1 s | DynamoDB (claimed) | "usually within one second or less", vendor statement | 2023 | read consistency doc |
| Follower-read staleness floor | ~5 s | CockroachDB | Closed timestamp "roughly trails real time by five seconds" | 2018 | follower reads RFC |
| Stale-read incident duration | 37 h 12 m | GitLab.com | Severity 2; ~150 k Sidekiq jobs affected | 2024-10 | incident review |
| Amplification outages | 86% / 89% traffic | GitLab.com | Two severity 1s, 15-18 min impact each, two days apart | 2025-10 | incident review |
Measured: the GitLab incident figures and every constant read out of source code. Claimed: DynamoDB's sub-second convergence, a vendor statement with no published distribution behind it, and CockroachDB's five seconds, which is a design target from its own RFC. Unverifiable this session: the widely cited SOSP 2015 Facebook study, which per its abstract measured on the order of 0.0004% of sampled reads returning results a linearizable system would forbid; the paper's hosts were unreachable under this session's network policy, so treat that number as reported folklore until you fetch it yourself. No source in this corpus publishes a replication-lag distribution for a large fleet; that gap is real, and it is why every system above ships a health filter rather than an assumption.
Every source behind this page, graded. This session's network policy allowed github.com and gitlab.com only, so the wall is repository record end to end: incident reviews, source files, RFCs, issues and in-repo vendor docs. Engineering-blog, paper and talk tiers are absent because their hosts were unreachable, not because the material was judged and cut; the hunt section says where those layers live.
The definitive stale-read incident write-up: misrouted Sidekiq reads, 37 hours, repeated premature mitigation, root cause found only through a per-worker replica-read metric. Includes the linked incident issue (#18676) and the revert MR (168630).
A six-step cascade from replica CPU saturation to site-wide pool exhaustion, twice in three days, with the load balancer's primary-fallback named as an amplifier in the lessons learned.
The canonical replication-lag disaster, read from the blog's markdown source at a pinned 2017 commit: lag broke replication entirely, and the manual re-sync procedure destroyed the primary's data directory.
The live incident record behind review #18681: three rounds of investigation, two
red-herring subsystems, and the note where a graph of replica reads by
data_consistency: always workers cracked the case.
The production sticking layer: per-client LSN in Redis with a 30-second TTL, and Lua scripts that parse and compare LSNs atomically so racing requests cannot regress the stored position.
The operational envelope around the code: 30-second primary sticking released early when replicas catch up, and lag thresholds (8 MB, 60 s) that evict replicas from the candidate pool.
The asynchronous half of the design: jobs enqueue with the primary's LSN;
:sticky and :delayed workers get replicas guaranteed caught up
to enqueue time, or a retry, or the primary. Every worker must declare a mode, enforced
by RuboCop.
The PR that upstreamed GitHub's approach into Rails: two-second post-write window per session, with the author noting GitHub's production variant runs five seconds plus a replication-delay circuit breaker, and a reviewer flagging the query cache as a remaining stale-read hole.
SEND_TO_REPLICA_DELAY = 2.seconds, a timestamp in
session[:last_write], and a comparison. The whole shipped mechanism is
thirty lines.
A five-year attempt to adopt Rails' replica switching in a mature application, blocked by a framework migrations bug and dependency gaps, closed with "it's probably easier to start afresh, rather than rebase this PR." Its predecessor (#2634) has been open since 2020.
The oldest complete implementation in the corpus, documented like a design paper in its header: client-ID-keyed positions, a ten-second cookie window sized to cross-DC replication, a one-millisecond store budget, and an explicitly accepted degraded mode.
The complete GTID-based design: MySQL returns the GTID in the OK packet, VTGate
wraps it, replicas run WAIT_FOR_EXECUTED_GTID_SET. Recommends failing on
timeout rather than escalating to the primary. Open since 2020; the user ask (#4718)
predates it by 18 months.
A maintainer-filed bug: with the source unreachable, seconds_behind_master
is NULL, was coerced to 0, and unboundedly stale replicas passed health checks. Fixed in
PR #9308.
Why the proxy cannot always save you: ProxySQL's GTID causal reads require the
server's session_track_gtids, which MariaDB did not provide. The guarantee
is only as available as the weakest layer's cooperation.
Seven modes spanning session, service and cluster causality, wait-based and route-based variants, a 10-second budget, primary retry on timeout, and honest warnings about the fast modes collapsing onto the primary under writes.
The strong-by-default database's opt-out: closed timestamps trailing real time by about five seconds let historical reads run on followers, "away from foreground traffic".
Generation two: the reader states a staleness budget
(with_max_staleness) and the system minimises staleness inside it, "more
tolerant to replication lag" than a fixed historical timestamp.
The token pipeline as a formal spec: every server response carries an
operationTime; drivers must store it in the session and replay it as
afterClusterTime on subsequent reads, without validating it.
The 2025 incarnation: replicas "may be arbitrarily out of date", sessions guarantee
sequential consistency via bookmarks, and the application carries the bookmark between
requests itself, shown as an x-d1-bookmark header.
Weak-by-default with a priced opt-in: eventually consistent unless
ConsistentRead is set; the strong read costs twice the capacity, is
unavailable on global secondary indexes, and may return a 500 during network trouble.
The write-side complement: a raft-replicated service that throttles bulk writers when replica lag crosses a threshold, keeping replicas valid for readers; its README names "mitigating write-then-read pains of master reads" as a use.
Three landings: a 2020 revert, pg_wal_replay_wait() committed 2024-04-02
and reverted 2024-04-11 (v17 shipped without it), and WAIT FOR committed
2025-11-05 for v19, with a recovery-deadlock fix as late as 2026-09-12.
The pattern reinvented at the edge for SQLite: a __txid cookie, replicas
polled every millisecond up to five seconds, cookie expiring after five minutes, writes
forwarded to the primary.
The counter-default for contrast: "etcd ensures linearizability for all other operations by default." A coordination store sells agreement, so it pays the read-time cost up front and lets callers opt down instead.
Six rungs from reproducing the anomaly to instrumenting the two silent failure modes. Two Postgres containers and any web framework are enough for all of them.
Stand up a primary and a streaming replica. Write a row, read it back from the
replica in a tight loop, and log the misses. Then pause replay on the replica
(SELECT pg_wal_replay_pause()) and watch every read go stale.
Done when: you can produce and clear a read-your-writes violation at will. Teaches: the anomaly is a race you can win or lose deliberately, not weather.
Add Rails-style switching: after a write, pin that session to the primary for two seconds. Now hold replay paused for ten seconds and measure the violation rate.
Done when: violations vanish under normal lag and return the moment lag exceeds the window. Teaches: a timer is a bet on the lag distribution's tail, and you now know exactly when it loses.
After each write, capture pg_current_wal_insert_lsn() into a cookie.
Before each replica read, compare against
pg_last_wal_replay_lsn() and poll (or use WAIT FOR on
PostgreSQL 19) with a one-second budget before giving up.
Done when: zero violations at any induced lag, and you can report added read latency at p50/p99. Teaches: the full mint-carry-route-wait pipeline and what the wait costs.
On wait timeout, fall back to the primary, but through a counter that allows at most N fallback reads per second and fails the rest. Load-test with replay paused fleet-wide.
Done when: total primary read load stays bounded no matter how many clients time out. Teaches: the October 2025 lesson, that an uncapped fallback is a self-aimed stampede.
Add a job queue. Enqueue each job with the LSN captured at enqueue time; workers
check the replica against the job's LSN and defer once (re-enqueue with delay) before
promoting to the primary, GitLab's :delayed semantics.
Done when: a write-then-enqueue-then-read chain never observes pre-write state and touches the primary only on double misses. Teaches: cross-actor causality, the case session tokens cannot cover.
Break your own router (route one "must-primary" endpoint to replicas) and your own health probe (make the lag query return NULL, coerced to 0). Add the two alerts that catch both: primary-declared callers observed on replicas, and replicas whose lag is unknown rather than low.
Done when: both injected faults page you before any user-visible symptom. Teaches: failure classes A and C from section 04, which no default dashboard shows.
The queries that found this material, grouped by what they surface. The first group works even from a network position that can only reach the big code forges, which is how this page was built.
site:gitlab.com gl-infra production "incident review" replicationgitlab.com/api/v4/projects/gitlab-com%2Fgl-infra%2Fproduction/issues?labels=incident-review&search=replication"stale read" OR "stale data" incident review database"load balancing" "shifted traffic to the primary""read-after-write" OR "read your writes" replica postmortem"causal_reads" OR "MASTER_GTID_WAIT" site:github.com"WAIT_FOR_EXECUTED_GTID_SET" proxy OR vtgate"session_track_gtids" causal"chronology protector" OR "cpPosIndex" mediawikigithub.com/postgres/postgres/commits/master/src/backend/commands/wait.cpg_wal_replay_wait revert pgsql-hackersrepo:vitessio/vitess "read after write" is:issue"data_consistency" sticky delayed sidekiq site:gitlab.com"seconds_behind_master" NULL healthcheck staleTerry "session guarantees" 1994 Bayou "read your writes""existential consistency" facebook sosp 2015 measurement"scaling memcache" nsdi 2013 "remote marker"intitle:"read your writes" engineering blog replica lag