Evidence ledger
24 sources
Checked 24 Sep 2026
Evidence ledger
One row per claim in Your write hasn't happened here yet: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
One row per claim. Every URL was fetched during the research session on 2026-09-24. The research environment's outbound network policy allowed github.com, raw.githubusercontent.com and gitlab.com; engineering blogs, documentation sites, paper archives and video hosts were unreachable. The guide is therefore deliberately built from the repository record: incident reviews on GitLab's public production tracker, source code, issues, pull requests, RFCs and in-repo documentation. Where a canonical paper is named in the prose (Terry et al. 1994; the SOSP 2015 Facebook measurement study), it is explicitly marked as not re-verifiable in this session and no linked evidence card is built on it. That limit is stated in the guide.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | GitLab | Incident Review: 2024-10-08: Pipelines not completing | postmortem | 2024-10 | 2026-09-24 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18681 | A refactor silently broke primary-sticking in Sidekiq middleware; the only symptom of the stale reads was stuck CI pipelines and merge requests. | "sidekiq middleware incorrectly routed certain queries to a replica db that should have been sent to the primary db, causing those queries to see slightly stale data (missing very recently committed transactions) ... a stale read bug." Severity::2; Start 2024-10-07 20:50 UTC, End 2024-10-09 10:02 UTC, "37h 12m"; "Estimated 150k sidekiq jobs (up to 135M)" affected. |
| 2 | GitLab | Incident Review: 2024-10-08 (detection difficulty) | postmortem | 2024-10 | 2026-09-24 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18681 | Staleness raises no alarm: the incident was closed as mitigated repeatedly while the actual bug persisted. | "The incident was marked as mitigated and reverted to active a few times due to the lack of a clear indicator of the problem"; root cause finally found via the gitlab_transaction_db_replica_count_total metric filtered to workers that should never read replicas. |
| 3 | GitLab | Incident Review: 2024-10-08 (sticking described) | postmortem | 2024-10 | 2026-09-24 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18681 | GitLab's normal behaviour: replicas serve reads until a write happens, then the session sticks to the primary. | "gitlab.com on a whole was able to operate (albeit degraded) with :always workers reading from replicas (until a write happens, then it sticks to the primary for subsequent reads)." |
| 4 | GitLab | 2024-10-08: Pipelines not completing and Stuck Merge Requests (incident issue) | postmortem | 2024-10 | 2026-09-24 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 | The user-facing symptom of a stale read in a workflow engine is a permanently wedged state machine, not an error. | Incident issue behind review #18681; pipelines "perpetually stuck in the running status after its last CI job finished"; merge requests stuck at "Your merge request is almost ready!" (quoted in the review). |
| 5 | GitLab | MR 168630: Revert "Merge branch 'sc1-session-map' into 'master'" | source | 2024-10-09 | 2026-09-24 | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/168630 | The fix for the 37-hour stale-read incident was a straight revert of the session-map refactor. | Title: Revert "Merge branch 'sc1-session-map' into 'master'"; state merged; merged_at 2024-10-09T09:57Z (GitLab API). |
| 6 | GitLab | Incident Review: GitLab.com slow and not loading some elements (INC-4589 / INC-4641) | postmortem | 2025-10 | 2026-09-24 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20692 | Falling back to the primary when replicas lag converts a replica-side overload into a site-wide outage. | Cascade as numbered in the review: "Replica Saturation ... Replication Lag: As replicas became CPU-bound, replication lag increased ... Load Balancer Failover: Rails database load balancing detected slow replica responses and shifted traffic to the primary node ... Primary Saturation ... Connection Pool Exhaustion ... Platform Impact". |
| 7 | GitLab | Incident Review INC-4589/INC-4641 (lesson) | postmortem | 2025-10 | 2026-09-24 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20692 | GitLab's own stated lesson names the amplification. | "Database Load Balancing Can Amplify Issues: When replicas fail health checks, shifting all traffic to the primary can accelerate rather than mitigate incidents." Two Severity 1 incidents (2025-10-07, 2025-10-09); "86% of all traffic dropped" and "89% of all traffic dropped"; trigger: "approximately 4,000 requests per minute" from one user; "Individual queries consumed 7-8 seconds of CPU time"; the query "joins 16 tables". |
| 8 | GitLab | Postmortem of database outage of January 31 (2017), blog source at pinned commit | postmortem | 2017-02-10 | 2026-09-24 | https://gitlab.com/gitlab-com/www-gitlab-com/-/raw/026a78dde923abd94a7b57640a1fddfebe07cb84/source/posts/2017-02-10-postmortem-of-database-outage-of-january-31.html.md | Replication lag is also an operational hazard: repairing it under pressure is what destroyed the primary's data. | "Due to the increased load, our PostgreSQL secondary's replication process started to lag behind. The replication failed as WAL segments needed by the secondary were already removed from the primary." Re-sync "involves removing the existing data directory on the secondary"; the removal was then executed on db1 (primary). Data lost 17:20-00:00 UTC; "roughly 5,000 projects, 5,000 comments and 700 new user accounts". |
| 9 | GitLab | lib/gitlab/database/load_balancing/sticking.rb | source | checked at master | 2026-09-24 | https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/load_balancing/sticking.rb | The production sticking implementation: per-client WAL positions in Redis, 30-second expiry, atomic LSN comparison in Lua. | EXPIRATION = 30 ("The number of seconds after which a session should stop reading from the primary"); Lua ATOMIC_STICK_SCRIPT parses LSNs ("0/16B3A78") into high/low pairs and only overwrites the stored position if the new LSN is greater. |
| 10 | GitLab | Database Load Balancing (admin documentation, in-repo) | vendor | checked at master | 2026-09-24 | https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/postgresql/database_load_balancing.md | The operational envelope: sticky reads expire at 30 s or when replicas catch up; lagging hosts are removed from the pool. | "After a write, GitLab temporarily routes reads for that user to the primary for up to 30 seconds. GitLab reverts to using replicas as soon as they have caught up, rather than always waiting the full 30 seconds." Defaults: max_replication_difference 8 MB, max_replication_lag_time 60 seconds; a lagging secondary "is skipped until it catches up". |
| 11 | GitLab | Sidekiq worker attributes: data_consistency (developer documentation, in-repo) | vendor | checked at master | 2026-09-24 | https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/sidekiq/worker_attributes.md | For asynchronous work the causality token travels with the job: jobs are enqueued with the primary's LSN and are delayed, retried or promoted to the primary until a replica catches up. Every worker must declare a choice, enforced by lint. | "jobs are now enqueued along with the current database LSN, the replica (for :sticky or :delayed) is guaranteed to be caught up to that point, or the job will be retried, or use the primary." :always is "Strongly discouraged"; the data_consistency field "is enforced by a RuboCop rule". |
| 12 | Rails core / GitHub | rails/rails PR #35073: Adds basic automatic database switching to Rails | source | merged 2019-01-30 | 2026-09-24 | https://github.com/rails/rails/pull/35073 | The framework default is a time bet, and GitHub's production version adds a lag-aware circuit breaker; the query cache was a known stale-read hole at merge time. | Author (eileencodes): "At GitHub in addition to the a 5 second delay we have a curcuit breaker that checks the replication delay and will send the query to a replica before the 5 seconds has passed" (sic). Reviewer (kamipo): after a write "the query cache on the :reading connection is still remained"; author: applications must "care about that case by themselves for now". |
| 13 | Rails core | activerecord database_selector resolver (source, main branch) | source | checked 2026-09-24 | 2026-09-24 | https://github.com/rails/rails/blob/main/activerecord/lib/active_record/middleware/database_selector/resolver.rb | The shipped default: after any write, all of that browser session's reads go to the primary for two seconds. | SEND_TO_REPLICA_DELAY = 2.seconds; reads go to the replica only when Time.now - context.last_write_timestamp >= send_to_replica_delay; the timestamp is stored in request.session[:last_write] (resolver/session.rb). |
| 14 | OpenStreetMap | openstreetmap-website PR #3201: WIP: Use database replicas for read requests | source | opened 2021-05-19, closed unmerged 2026-03-18 | 2026-09-24 | https://github.com/openstreetmap/openstreetmap-website/pull/3201 | Adopting replica reads in a mature Rails application stalled for five years on framework edge cases and was closed unmerged. | "the resolver will continue to use the same (write) connection for two seconds for subsequent reads"; blocked because "there's a bug in rails when running migrations so we can't merge this yet" (rails#41855); closed with "it's probably easier to start afresh, rather than rebase this PR." |
| 15 | OpenStreetMap | openstreetmap-website PR #2634: Use database replicas for read requests | source | opened 2020-05-27, still open | 2026-09-24 | https://github.com/openstreetmap/openstreetmap-website/pull/2634 | First attempt at the same change, held for years on test-suite integration. | "until the problem with the test suite is sorted, this is on hold"; "I've reported the problem I found to the rails team". |
| 16 | Wikimedia | ChronologyProtector.php (MediaWiki core, GitHub mirror) | source | checked at master | 2026-09-24 | https://github.com/wikimedia/mediawiki/blob/master/includes/libs/Rdbms/ChronologyProtector.php | The oldest complete implementation of the pattern: position stored under a client ID, ten-second cookie window sized to cross-DC replication, explicit storage requirements and an explicit degraded mode. | "temporarily remembering the database positions after a client makes a write, and on their next web request we will prefer non-lagged database replicas"; cookies "UseDC=master and cpPosIndex=<posIndex>@<write time>#<clientId>"; "The ten seconds window is meant to account for the time needed for the database writes to have replicated across all active database replicas, including the cross-dc latency"; store must respond "within the order of one millisecond"; on data loss "users may be temporary confused as they observe their own actions as not being immediately reflected." |
| 17 | Vitess (CNCF) | RFC: Read After Write (issue #6843) | adr | 2020-10-09, open | 2026-09-24 | https://github.com/vitessio/vitess/issues/6843 | The designed mechanism (GTID returned to client, WAIT_FOR_EXECUTED_GTID_SET on replica) and the deliberate choice to fail on timeout rather than escalate to the primary. | Proposal: MySQL "returns GTIDs for each update in the OK packet"; VTGate issues WAIT_FOR_EXECUTED_GTID_SET on the replica; on timeout, "should the query be sent to primary or still just get executed on that replica?", recommending failing the query to prevent cascading load. Open since 2020. |
| 18 | Vitess (CNCF) | causal consistent read support from vitess replica (issue #4718) | source | 2019-03-12, open | 2026-09-24 | https://github.com/vitessio/vitess/issues/4718 | The user ask predates the RFC by 18 months and names the resource-exhaustion risk of blocking waits. | Requests routing "to replica tablet type if and only if the replica has executed at least or beyond the given GTID"; notes MySQL's wait function "could have a blocking effect, and causing exhaustion of resources if all queries got stuck waiting." |
| 19 | Vitess (CNCF) | vtgate serves queries from replicas when its source is unavailable and replication is unhealthy (issue #9307) | source | 2021-12-01, closed (fix PR #9308) | 2026-09-24 | https://github.com/vitessio/vitess/issues/9307 | A NULL lag signal was coerced to zero, so unhealthily lagging replicas passed health checks and served unboundedly stale reads. | When the source is unreachable MySQL reports seconds_behind_master as NULL; the system "silently converts this to 0", producing "replicas serving VERY stale reads that can in turn cause a cascade of downstream issues." |
| 20 | ProxySQL | Causal Consistency Reads & Tracking GTID on MariaDB/Galera (issue #2134) | source | 2019-07-12, closed | 2026-09-24 | https://github.com/sysown/proxysql/issues/2134 | Proxy-level causal reads depend on a server feature; where the server lacks session GTID tracking, the proxy cannot provide the guarantee. | "MariaDB Server does not support session_track_gtids", the feature ProxySQL's GTID causal-read tracking on MySQL 5.7 depends on. |
| 21 | MariaDB Corporation | MaxScale ReadWriteSplit router documentation, causal_reads (24.02 branch) | vendor | checked 2026-09-24 | 2026-09-24 | https://github.com/mariadb-corporation/MaxScale/blob/24.02/Documentation/Routers/ReadWriteSplit.md | The proxy's complete decision space in one enum: session/service/cluster causality, wait-based and route-based variants, a 10-second wait budget, and primary retry as the escape hatch. | Modes none, local, global, fast, fast_global, universal, fast_universal; "If the replica has not caught up to the primary within the configured time ... it will be retried on the primary"; causal_reads_timeout default "10s"; universal mode: latency "increases by roughly twice the network latency between MaxScale and the database cluster"; fast modes: "the traffic will end up being routed almost exclusively to the primary server" under write-heavy load. Based on MASTER_GTID_WAIT. |
| 22 | Cockroach Labs | RFC 20180603: Follower reads | adr | 2018-06 | 2026-09-24 | https://raw.githubusercontent.com/cockroachdb/cockroach/master/docs/RFCS/20180603_follower_reads.md | A database that gives every read a fresh answer offers staleness as an explicit, bounded opt-out; the closed timestamp trails real time by about five seconds. | "Consistent historical reads are useful for analytics queries and ... allow such queries to be carried out more efficiently, and, with appropriate configuration, away from foreground traffic"; the closed-timestamp target: "the min proposal timestamp roughly trails real time by five seconds." |
| 23 | Cockroach Labs | RFC 20210519: Bounded staleness reads | adr | 2021-05 | 2026-09-24 | https://raw.githubusercontent.com/cockroachdb/cockroach/master/docs/RFCS/20210519_bounded_staleness_reads.md | Second-generation design: let the reader state a staleness budget and have the system minimise staleness inside it, tolerant of variable replication lag. | Bounded staleness reads "use a dynamic, system-determined timestamp, subject to a user-provided staleness bound, to read nearby replicas while minimizing data staleness"; exposed as AS OF SYSTEM TIME with_max_staleness(INTERVAL). |
| 24 | MongoDB | Driver specification: Causal Consistency (mongodb/specifications) | adr | checked at master | 2026-09-24 | https://github.com/mongodb/specifications/blob/master/source/causal-consistency/causal-consistency.md | The driver-level token design: every server response carries an operationTime; the session replays it as afterClusterTime on later reads. | Read-your-writes defined as "an application can read its own writes"; drivers MUST save "the operationTime returned by servers for all operations" and pass it "in the afterClusterTime field of the readConcern field for subsequent causally consistent read and write operations"; "Drivers MUST NOT attempt to validate the supplied operationTime." |
| 25 | Cloudflare | D1 read replication documentation (cloudflare-docs, production branch) | vendor | checked 2026-09-24 | 2026-09-24 | https://github.com/cloudflare/cloudflare-docs/blob/production/src/content/docs/d1/best-practices/read-replication.mdx | The 2025 incarnation of the same token: sessions carry a bookmark, and the application must transport it between requests itself (shown via an x-d1-bookmark header). |
"at any given time, a read replica may be arbitrarily out of date"; "Sessions API ensures sequential consistency for all queries in a session"; the example reads the bookmark from request.headers.get("x-d1-bookmark") and sets it on the response. |
| 26 | AWS | DynamoDB developer guide: Read consistency (archived awsdocs repo) | vendor | checked 2026-09-24 | 2026-09-24 | https://github.com/awsdocs/amazon-dynamodb-developer-guide/blob/master/doc_source/HowItWorks.ReadConsistency.md | The managed-service default is eventual consistency, with the strong option carrying availability and latency caveats. | "DynamoDB uses eventually consistent reads, unless you specify otherwise"; replication completes "usually within one second or less"; strongly consistent reads "are not supported on global secondary indexes", "might not be available" during network delay (HTTP 500), and "may experience a higher latency". |
| 27 | AWS | DynamoDB developer guide: Read/write capacity mode (archived awsdocs repo) | vendor | checked 2026-09-24 | 2026-09-24 | https://github.com/awsdocs/amazon-dynamodb-developer-guide/blob/master/doc_source/HowItWorks.ReadWriteCapacityMode.md | The price of read-your-writes on demand: a strongly consistent read costs exactly twice an eventually consistent one. | "A strongly consistent read request of an item up to 4 KB requires one read request unit. An eventually consistent read request of an item up to 4 KB requires one-half read request unit." |
| 28 | GitHub | freno README (github/freno) | source | checked 2026-09-24 | 2026-09-24 | https://github.com/github/freno/blob/master/README.md | GitHub's lag-throttling service is also the read-side validity oracle: writers slow down so that replicas stay readable. | "freno will throttle cooperative clients when replication lag exceeds a pre-defined threshold"; "freno can also be used to determine actual lag to infer validity of replicas. This can assist in mitigating write-then-read pains of master reads." |
| 29 | PostgreSQL | Commit 06c418e: Implement pg_wal_replay_wait() stored procedure | source | 2024-04-02 | 2026-09-24 | https://github.com/postgres/postgres/commit/06c418e | The database-level wait primitive landed in core in April 2024 for the 17 cycle. | Commit by akorotkov, 2024-04-02: "Implement pg_wal_replay_wait() stored procedure"; waits for a WAL position to be replayed on a standby; cannot run inside an atomic context. |
| 30 | PostgreSQL | Commit 772faaf: Revert: Implement pg_wal_replay_wait() stored procedure | source | 2024-04-11 | 2026-09-24 | https://github.com/postgres/postgres/commit/772faaf | Nine days later the primitive was fully reverted out of the 17 cycle; PostgreSQL 17 shipped without it (src/backend/commands/waitlsn.c does not exist on REL_17_STABLE, checked 2026-09-24). |
Commit by akorotkov, 2024-04-11: "Revert: Implement pg_wal_replay_wait() stored procedure ... per review by Heikki Linnakangas." |
| 31 | PostgreSQL | Commit history of src/backend/commands/wait.c (master) | source | checked 2026-09-24 | 2026-09-24 | https://github.com/postgres/postgres/commits/master/src/backend/commands/wait.c | There were at least three landing attempts across six years: a 2020 implementation ("waiting for given lsn at transaction start", 0f5ca02f53) reverted in April 2020, the 2024 attempt reverted, and the 2025 command that stuck. | History shows a revert of 0f5ca02f53 dated April 8, 2020, then "Implement WAIT FOR command" (447aae13b03, akorotkov, 2025-11-05) in the v19 cycle, followed by option, grammar, timeline and overflow fixes into September 2026. |
| 32 | PostgreSQL | Commit e0c160c: Prevent WAIT FOR LSN from deadlocking recovery on held locks | source | 2026-09-12 | 2026-09-24 | https://github.com/postgres/postgres/commit/e0c160c307264654a2e91d582cc0935243bc9253 | The primitive was still receiving deadlock-prevention fixes twelve days before this guide was researched. | Commit by akorotkov, 2026-09-12: "Prevent WAIT FOR LSN from deadlocking recovery on held locks"; a backend waiting for replay can hold locks the startup process needs. |
| 33 | Fly.io | litefs http/proxy_server.go | source | checked at main | 2026-09-24 | https://github.com/superfly/litefs/blob/main/http/proxy_server.go | The same client-token design reinvented for edge SQLite in 2022: a transaction-ID cookie, millisecond polling on replicas, a five-second wait budget. | TXIDCookieName = "__txid"; DefaultPollTXIDInterval = 1 * time.Millisecond; DefaultPollTXIDTimeout = 5 * time.Second; DefaultCookieExpiry = 5 * time.Minute; comment: "handle primary redirection and TXID consistency on replicas." |
| 34 | etcd (CNCF) | etcd API guarantees (etcd-io/website) | vendor | checked at main | 2026-09-24 | https://github.com/etcd-io/website/blob/main/content/en/docs/v3.5/learning/api_guarantees.md | The opposite default exists: a coordination store makes every read linearizable unless the caller opts down. | "etcd ensures linearizability for all other operations by default." |
Named but not linked (unreachable under this session's egress policy)
- Terry, Demers, Petersen, Spreitzer, Theimer, Welch, "Session Guarantees for Weakly Consistent Replicated Data", PDIS 1994: the paper that named "read your writes" as one of four session guarantees, in the Bayou project at Xerox PARC. Identified via search-index metadata only; the PDF hosts were unreachable, so no claim in the guide relies on its contents beyond the name and date.
- Lu et al., "Existential Consistency: Measuring and Understanding Consistency at Facebook", SOSP 2015: reported (per the paper's abstract as surfaced by search indexing, not re-fetched) that about 0.0004% of sampled reads to Facebook's TAO would have returned different results under linearizability. Quoted in the guide with this caveat attached.
- Nishtala et al., "Scaling Memcache at Facebook", NSDI 2013: describes the "remote marker" mechanism for cross-region read-after-write; named in prose only, no claim built on it.