Data Platforms 21 Sep 2026 28 min read

The lock is brief. The queue is the outage.

How production teams change the schema of a hot relational table without stopping writes, and why the lock queue rather than the lock is what takes systems down.

Read the field guide → The evidence ledger

Reconstructs live schema change from the places fourteen organisations wrote their lessons down: GitLab's production incident tracker, gh-ost's design docs and bug reports, the CockroachDB and TiDB implementations of the F1 protocol, and the safety libraries GoCardless, Braintree, Doctolib and Instacart built after migrations bit them. A reader leaves able to name the lock-queue mechanism behind migration outages, choose between in-place DDL, shadow-table copy and schema-as-versioned-state, and set the timeouts that decide whether a bad migration costs customers or only the deploy queue.

The finding that surprised me

Where lock-timeout-and-retry is fully institutionalised the failure mode changes address: all four GitLab 2026 migration incidents are severity-3 deploy stalls with zero reported customer impact, and the terminal escalation after fifteen years of tooling is still Vitess's forced cut-over, which kills the competing queries.

What you get out of it

The outage mechanism needs no lock to be held: an ALTER TABLE waiting behind one long query walls off the table for every later query, which is why GoCardless (750 ms), GitLab (100 ms schedule) and Doctolib (5 s) all ship lock acquisition timeouts with retries.
A fail-closed lock guardrail converts customer outages into recurring severity-3 deploy chores; GitLab's 2026 incident record is the operating proof.
Trigger-based capture was rejected by GitHub in 2016 with production evidence (unthrottleable load, lock downs); binlog capture won, and Vitess has since moved capture into the database's own replication, de-recognising both external tools in v22.
The shadow-table copy path fails silently: INSERT IGNORE ate rows under a changed unique key (gh-ost #1526) and a semi-sync commit stall hid a row from both capture and copy (#1039), so checksumming is a component, not an option.
The F1 two-version protocol run by CockroachDB and TiDB and the expand-and-contract deploy folklore are the same invariant at different layers; pgroll makes the equivalence explicit by serving both schema versions through views.

Scope

Why this, now. GitLab.com logged four schema-migration incidents between April and September 2026, Meta archived its pioneering OnlineSchemaChange tool in August 2026, and Vitess v22 de-recognised gh-ost and pt-osc; the tooling generation that defined this practice is being retired while the failure mode it guards against is as live as ever.

What it does not cover. Resharding and database splitting, engine swaps and cross-store data migration (covered by the 2026-08-29, 2026-09-19 and 2026-09-20 digs), event and API schema evolution, and document stores; engineering blogs, papers and talks were unreachable from the research environment, which could reach only github.com and gitlab.com, and that constraint is disclosed in the page.

Open the field guide → Self-contained: it loads nothing at read time, follows your system theme, and prints cleanly.

Keep digging

Other field guides

Data Platforms

The lock is brief. The queue is the outage.

What you get out of it

Scope

Other field guides

When the database cannot be split: ten years of GitLab.com, read from its own incidents and design documents

Swapping the engine under the client: ten years of Pinterest, read from its own repositories

Own the layer above the engine: ten years of LinkedIn's data platform