Evidence ledger
24 sources
Checked 21 Sep 2026
Evidence ledger
One row per claim in The lock is brief. The queue is the outage.: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how production teams change the schema of a hot relational table without stopping writes, and why the lock queue, not the lock, is the thing that takes systems down.
Research date: 2026-09-21. All URLs fetched from this session's research environment on that
date. Environment constraint, disclosed in the guide: the research environment's network
policy allowed only github.com, raw.githubusercontent.com and gitlab.com. Engineering
blogs, papers hosted at vldb.org/ACM, conference talks and vendor documentation sites were
unreachable and are therefore absent from this ledger; repository-hosted design docs, incident
trackers, safety-library READMEs and release notes stand in for them. The F1 schema-change
paper is cited through the CockroachDB RFC and the TiDB design doc, both of which implement it
and were fetched.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | GitHub | gh-ost doc: Why triggerless? | adr | 2016 (doc, maintained) | 2026-09-21 | https://github.com/github/gh-ost/blob/master/doc/why-triggerless.md | Trigger-based migration load has caused production lockdowns and cannot be paused | "We have evidenced near or complete lock downs in production, to the effect of rendering the table or the entire database inaccessible due to lock contention." / "the tool may not, at any stage, cancel the triggers... even as the online operation throttles, the master is brought down by the load of the triggers." |
| 2 | GitHub | gh-ost doc: Why triggerless? | adr | 2016 | 2026-09-21 | https://github.com/github/gh-ost/blob/master/doc/why-triggerless.md | Trigger-based migrations cannot be reliably tested on replicas | "even with Statement Based Replication we do not get a reliable representation of the migration as it would have executed on the master" |
| 3 | GitHub | gh-ost doc: Cut-over step | adr | 2016 | 2026-09-21 | https://github.com/github/gh-ost/blob/master/doc/cut-over.md | The Facebook OSC cut-over is non-atomic and leaves a window with no table | "In between the two renames there's a brief period of time where your table just does not exist, and queries will fail." |
| 4 | GitHub | gh-ost issue #82: safe, blocking, atomic cut-over | adr | 2016-06-26 | 2026-09-21 | https://github.com/github/gh-ost/issues/82 | The atomic cut-over uses a sentry table and fails back to the original table | "We create tbl_old as a blocker for a premature swap" / "No matter what happens, at the end of operation we look for the ghost table. Is it still there? Then we know the operation failed, 'atomically'." |
| 5 | GitHub | gh-ost README | source | maintained, 2026 | 2026-09-21 | https://github.com/github/gh-ost/blob/master/README.md | gh-ost decouples migration write load and is continuously tested by migrating production tables on replicas | "True pause: when gh-ost throttles, it truly ceases writes on master" / "We have multiple replicas where we iterate our entire fleet of production tables, migrating them one by one, checksumming the results" |
| 6 | GitHub | gh-ost command-line flags doc | vendor | maintained, 2026 | 2026-09-21 | https://github.com/github/gh-ost/blob/master/doc/command-line-flags.md | Instant DDL is preferred but still needs a metadata lock; gh-ost falls back automatically | "--attempt-instant-ddl is disabled by default, but the risks of enabling it are relatively minor: gh-ost may need to acquire a metadata lock at the start of the operation... it could be a problem for users that start the DDL during a period with long running transactions." |
| 7 | GitHub | gh-ost main.go | source | maintained, 2026 | 2026-09-21 | https://github.com/github/gh-ost/blob/master/go/cmd/gh-ost/main.go | Defaults: chunk-size 1000 rows (10–100,000), max-lag-millis 1500, cut-over-lock-timeout-seconds 3, default-retries 60 | chunkSize := flag.Int64("chunk-size", 1000, ...) / maxLagMillis := flag.Int64("max-lag-millis", 1500, ...) / cutOverLockTimeoutSeconds := flag.Int64("cut-over-lock-timeout-seconds", 3, ...) |
| 8 | GitHub / community | gh-ost issue #1526: lose data with UNIQUE KEY | source | 2025-04-07 | 2026-09-21 | https://github.com/github/gh-ost/issues/1526 | Row copy via INSERT IGNORE silently drops rows that collide under a changed unique key/collation | "insert ignore into _temp_table_gho select * from temp_table lock in share mode; then the UNIQUE KEY for channel column have been ignore, then lose data." |
| 9 | GitHub / community | gh-ost issue #1039: data loss with semi-sync | source | 2021-10-26 | 2026-09-21 | https://github.com/github/gh-ost/issues/1039 | A transaction blocked in commit can be invisible both to the binlog listener and the row-copy boundary query | "The row with id=3297 has neither been notified to DMLListener nor copied to _gho table. It's lost." |
| 10 | GitHub / community | gh-ost PR #1698 (closed unmerged): fix cutover retries | source | 2026-06-04 | 2026-09-21 | https://github.com/github/gh-ost/pull/1698 | Cut-over timeout could deadlock the migration's own event pipeline; fix superseded by #1637 | Unbuffered channel send to allEventsUpToLockProcessed with no receiver hangs executeWriteFuncs; maintainer: "I believe the issue you describe was fixed by #1637." Closed without merge. |
| 11 | GoCardless | activerecord-safer_migrations README | source | 2016+, maintained | 2026-09-21 | https://github.com/gocardless/activerecord-safer_migrations/blob/master/README.md | The lock queue, not the lock duration, causes downtime; defaults lock_timeout 750 ms, statement_timeout 1500 ms | "Even if the lock is only held briefly, it will block all other access to the table while it is in the lock queue, as it conflicts with all other locks." / "The initial lock_timeout default is 750ms, and the initial statement_timeout default is 1500ms." |
| 12 | Instacart (ankane) | strong_migrations README | source | maintained, 2026 | 2026-09-21 | https://github.com/ankane/strong_migrations/blob/master/README.md | Dropping a column breaks the app because the ORM caches columns; safe path is ignore → deploy → drop | "Active Record caches database columns at runtime, so if you drop a column, it can cause exceptions until your app reboots." Battle-tested at Instacart. |
| 13 | Instacart (ankane) | strong_migrations README | source | maintained, 2026 | 2026-09-21 | https://github.com/ankane/strong_migrations/blob/master/README.md | Backfilling in the migration transaction holds the lock for the whole backfill; safe backfill = batching, throttling, outside a transaction | "backfilling in the same transaction that alters a table keeps the table locked for the duration of the backfill" / "There are three keys to backfilling safely: batching, throttling, and running it outside a transaction." |
| 14 | Braintree / PayPal | pg_ha_migrations README | adr | maintained, 2026 | 2026-09-21 | https://github.com/braintree/pg_ha_migrations/blob/master/README.md | Schema rollback is rejected as a strategy; roll forward instead | "we require that 'Rollback strategies do not involve reverting the database schema to its previous version'... In the rare case where we need to 'undo' a schema change, we roll forward, rather than rolling back" |
| 15 | Braintree / PayPal | pg_ha_migrations README | adr | maintained, 2026 | 2026-09-21 | https://github.com/braintree/pg_ha_migrations/blob/master/README.md | Safety is made explicit rather than magical: safe_/unsafe_/raw_ prefixes; two concern classes | "Database safety (e.g., long-held locks)" and "Application safety (e.g., dropping columns the app uses)" / "it's better to understand exactly what the database is doing so that... operations are not a surprise during your deploy cycle" |
| 16 | Doctolib | safe-pg-migrations README | source | maintained, 2026 | 2026-09-21 | https://github.com/doctolib/safe-pg-migrations/blob/master/README.md | Foreign keys and check constraints are split into add-without-validation then validate; autovacuum can block DDL | "Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint." / "A vacuum / autovacuum is running on the table, holding a ShareUpdateExclusiveLock, you are most likely out of luck for the current migration." |
| 17 | GitLab | doc: Avoiding downtime in migrations | adr | maintained, 2026 | 2026-09-21 | https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/database/avoiding_downtime_in_migrations.md | Dropping a column is a three-release protocol (ignore in M, drop in M+1, remove rule in M+2); FK removal locks in reverse order to avoid deadlock | "with_lock_retries runs the removal in a transaction with a short lock_timeout and retries when it cannot acquire the lock, instead of blocking until a timeout." / "remove_foreign_key_if_exists defaults to reverse_lock_order: true, which locks the referenced table before the source table to prevent deadlocks" |
| 18 | GitLab | with_lock_retries.rb | source | maintained, 2026 | 2026-09-21 | https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/with_lock_retries.rb | The retry schedule starts at lock_timeout 100 ms and is bounded; worst case about 40 minutes | "First element: DB lock_timeout... Worst case, this configuration would retry for about 40 minutes." First iterations: [0.1.seconds, 0.05.seconds]. |
| 19 | GitLab | doc: Large tables limitations | adr | maintained, 2026 | 2026-09-21 | https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/database/large_tables_limitations.md | Hard size limits gate schema changes on GitLab.com: no new index above 50 GB, no new column above 100 GB; goal is all tables under 100 GB | "Cannot add an index — 50 GB; Cannot add a column with foreign key — 50 GB; Cannot add a new column — 100 GB" |
| 20 | GitLab | doc: Batched background migrations | vendor | maintained, 2026 | 2026-09-21 | https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/database/batched_background_migrations.md | Backfills have caused incidents; they are throttled on WAL backlog, autovacuum activity and apdex, pausing 10 minutes on a stop signal | "there have been incidents due to the heavy load from these migrations while the database was underperforming, a throttling mechanism exists to mitigate future incidents" |
| 21 | GitLab | Production incident #22699 | postmortem | 2026-08-13 | 2026-09-21 | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22699 | ACCESS EXCLUSIVE on projects+deployments exhausted lock retries twice; impact was blocked deployments, not customer downtime | "The post-deploy migration needed an ACCESS EXCLUSIVE lock on the projects and deployments tables, and it exhausted its lock-retry attempts twice." Severity 3. |
| 22 | GitLab | Production incident #22849 | postmortem | 2026-09-03 | 2026-09-21 | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22849 | Same failure class; retry per runbook succeeded; no customer-facing impact | "The failed post-deploy migration blocked deployment progression. No customer-facing service impact has been identified." Severity 3. |
| 23 | GitLab | Production incident #22873 | postmortem | 2026-09-07 | 2026-09-21 | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22873 | An autovacuum on the notes table blocked trigger creation; migration skipped and rescheduled | "An ongoing autovacuum on the notes table is causing lock contention during the migration." Severity 3. |
| 24 | GitLab | Production incident #21712 | postmortem | 2026-04-06 | 2026-09-21 | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21712 | Adding a FK to a partitioned CI table deadlocked against application queries under high traffic | "The deadlock was triggered by locking conflicts with application queries during high traffic." Severity 3, no customer-facing impact reported. |
| 25 | PostgreSQL | ALTER TABLE reference (alter_table.sgml) | vendor | maintained, 2026 | 2026-09-21 | https://github.com/postgres/postgres/blob/master/doc/src/sgml/ref/alter_table.sgml | ACCESS EXCLUSIVE is the default lock for ALTER TABLE subforms; ADD FOREIGN KEY takes SHARE ROW EXCLUSIVE on both tables | "An ACCESS EXCLUSIVE lock is acquired unless explicitly noted." / "ADD FOREIGN KEY requires only a SHARE ROW EXCLUSIVE lock... also acquires a SHARE ROW EXCLUSIVE lock on the referenced table" |
| 26 | CockroachDB | RFC 20151014_online_schema_change.md | adr | 2015-10-14 | 2026-09-21 | https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/20151014_online_schema_change.md | The F1 protocol: DELETE_ONLY → WRITE_ONLY → PUBLIC with at most two active descriptor versions; backfill runs as small transactions | "It is invalid to jump from the DELETE_ONLY to PUBLIC state or vice versa... a cluster will not create invalid data when two consecutive versions of the descriptor are in use concurrently." References "Online, Asynchronous Schema Change in F1" (Rae et al., VLDB 2013). |
| 27 | PingCAP | TiDB design doc: online DDL (2018-10-08) | adr | 2018-10-08 | 2026-09-21 | https://github.com/pingcap/tidb/blob/master/docs/design/2018-10-08-online-DDL.md | Same two-version invariant, independently implemented: one owner steps the cluster through none → delete only → write only → write reorg → public | "At the same time, there are at most two different versions of the schema on the same table for all nodes of the system." |
| 28 | Percona | pt-online-schema-change (tool source, POD docs) | source | maintained, 2026 | 2026-09-21 | https://github.com/percona/percona-toolkit/blob/3.x/bin/pt-online-schema-change | Trigger-based copy; chunked with target chunk-time 0.5 s, max-lag 1 s default; foreign keys break the atomic rename | "Any modifications to data in the original tables during the copy will be reflected in the new table, because the tool creates triggers" / "The technique of atomically renaming the original and new tables does not work when foreign keys refer to the table." / chunk-time "type: float; default: 0.5", max-lag "type: time; default: 1s" |
| 29 | Meta (Facebook) | OnlineSchemaChange repository | source | 2017; archived 2026-08-04 | 2026-09-21 | https://github.com/facebookincubator/OnlineSchemaChange | The 2010/2017 OSC lineage (copy mode, table swap) is now archived; repository read-only since August 2026 | Repo page: "a tool for making schema changes for MySQL tables in a non-blocking way"; archived 2026-08-04. |
| 30 | SoundCloud | Large Hadron Migrator README | source | 2011+; archived | 2026-09-21 | https://github.com/soundcloud/lhm/blob/master/README.md | The original problem statement: locking ALTER TABLE took sites down for an hour+; LHM now unmaintained | "the locking nature of ALTER TABLE may take your site down for an hour or more while critical tables are migrated" / "This repository is no longer actively maintained." |
| 31 | Vitess | v20.0.0 release notes | source | 2024 | 2026-09-21 | https://github.com/vitessio/vitess/blob/main/changelog/20.0/20.0.0/release_notes.md | Vitess stopped embedding gh-ost | "Vitess 20.0 drops support for gh-ost DDL strategy. vttablet binary no longer embeds a gh-ost binary." |
| 32 | Vitess | v22.0.0 release notes | source | 2025 | 2026-09-21 | https://github.com/vitessio/vitess/blob/main/changelog/22.0/22.0.0/release_notes.md | gh-ost and pt-osc are no longer recognized; replication-native "vitess" strategy is the recommended way | "Vitess no longer recognizes the gh-ost and pt-osc (pt-online-schema-change) Online DDL strategies. The vitess strategy is the recommended way to make schema changes at scale." |
| 33 | Vitess | PR #14546: cut-over backoff and forced cut-over | source | merged 2023-12-13 | 2026-09-21 | https://github.com/vitessio/vitess/pull/14546 | Failed cut-overs retried immediately compound load; backoff escalates 1→5→10→30 min; forced cut-over kills competing queries | "the database, that is already under heavy load, needs to cope with frequently recurring cut-over attempts, which themselves put additional locks on tables"; forced cut-over "brutally kills any queries using the migrated table, as well as terminating any transactions that are holding locks on the table". MySQL 8.0 only (needs performance_schema.data_locks). |
| 34 | Xata | pgroll README | source | 2023+, maintained | 2026-09-21 | https://github.com/xataio/pgroll/blob/main/README.md | Serving both schema versions at once via views makes rollback instant and decouples app rollout | "safe and reversible schema migrations for PostgreSQL by serving multiple schema versions simultaneously" / "works by creating virtual schemas by using views on top of the physical tables"; clients select a version with SET search_path. |
Category absences (findings, not omissions)
- Papers: the F1 paper (Rae et al., PVLDB 6(11), 2013) is the intellectual root of every distributed implementation here, but vldb.org and ACM were unreachable from this environment. It is cited through two independent implementations (rows 26, 27) that quote its state machine.
- Talks: no conference-talk hosting site was reachable. Shlomi Noach's gh-ost and Vitess Online DDL talks exist (speakerdeck.com); the same author's design writing is captured in rows 1–4 and 33 instead.
- Engineering blog posts: GoCardless's "Zero-downtime Postgres migrations — the hard parts", Braintree's "PostgreSQL at Scale" and the GitHub gh-ost announcement post are the canonical blog accounts; each is represented here by the repository artifact its team built from the lesson (rows 11, 14–15, 1–5), which encode the same claims in enforceable form.
- GitHub availability reports (github.blog) describe at least three schema-migration incidents (April 2025, August 2025, May 2026) per search-result summaries, but the pages were unreachable and are therefore not cited in the guide.