advanced 2 min answer

A team deploys many times a day and must change the database schema. What discipline makes this safe?

migrationsexpand-contractrollbackbackfillnotiondesign
Show the full answer Hide the answer

The discipline

Expand and contract, always — because a rolling deployment means old and new application versions run simultaneously against one database, and any schema change only one of them tolerates is an outage.

1. Expand. Add the new structure in a backward-compatible way — nullable column, new table, additional index. Old code is unaffected. Note that adding a column with a default may rewrite the table on some engines, which is the difference between seconds and hours.

2. Deploy code writing both. Every write path — including batch jobs, workers, imports and admin tooling. Missing one is the most common defect, and it produces rows correct in one column and wrong in the other.

3. Backfill in rate-limited, resumable batches, monitored against database load with the ability to pause. An unthrottled backfill on a large hot table is a production incident.

4. Dual-read with comparison. Read both, use the old, log discrepancies until the rate reaches zero. This converts correctness from a hope into a measurement and catches the write path missed in step two.

5. Switch reads behind a feature flag, so cutover and rollback are configuration changes rather than deployments.

6. Stop writing the old, in a separate release. 7. Drop it, in a further one.

The case most often forgotten

Rollback. Dropping the old structure before every deployed version has stopped using it — including a version you might roll back to — breaks the rollback path. The contract step must wait for that, not merely for the current version.

The enforcement that makes it stick

Migration linting in CI: no blocking DDL on large tables, no destructive change without a deprecation period, no NOT NULL addition without a default and a backfill.

Reviews catch what people remember; pipelines catch what they do not — and most migration tools will happily generate a single-step destructive change.

The failure to watch for

The permanent expand, where dual-write runs for years because nobody scheduled the contract. The schema accumulates half-finished migrations and the dual-write code becomes load-bearing.