advanced 2 min answer

A platform deploying many times a day must change a column type on a very large table with no maintenance window. Walk through the sequence.

zero-downtimeexpand-contractbackfillschemagithubdesign
Show the full answer Hide the answer

The sequence

Expand and contract, because a rolling deployment means old and new application versions run simultaneously against one database, and any schema change only one of them tolerates is an outage.

1. Add the new column, nullable. Backward compatible; old code is unaffected. On a very large table, adding a nullable column without a default is fast in modern engines — adding one with a default may rewrite the table, which is the difference between seconds and hours.

2. Deploy code that writes both columns. Every write path — including batch jobs, background workers, imports and admin tooling. Missing one is the most common defect, and it produces rows that are correct in one column and wrong in the other.

3. Backfill in rate-limited batches. Small batches with a pause between them, monitored against database load, and resumable. An unthrottled backfill on a large hot table is a production incident.

4. Dual-read with comparison. Read both, use the old, log discrepancies. This converts the migration's correctness from a hope into a measurement and catches the write path that was missed in step two.

5. Switch reads to the new column, behind a feature flag so cutover and rollback are configuration changes rather than deployments.

6. Stop writing the old column, in a separate release.

7. Drop the old column, in a further separate release, after every deployed version has stopped using it — including long-running batch jobs, offline workers, and any release that might be rolled back to.

What to monitor between steps

  • Database load during backfill, with the ability to pause.
  • Discrepancy rate from the dual-read comparison, which should reach zero before proceeding.
  • Error rates on every write path, especially the ones that are exercised rarely.

Where this goes wrong

Contracting too early, before every deployed version has stopped using the old column. The rollback case is the one most often forgotten: if a release is rolled back after the column is dropped, the rolled-back version fails.

The permanent expand, where dual-write is left running for years because nobody scheduled the contract, so the schema accumulates half-finished migrations and the dual-write code becomes load-bearing.

Forgetting non-application writers — reporting tools, ETL jobs, direct support corrections.

Assuming the migration tool handles it. Most will happily generate a single-step destructive change, which is why migration linting in CI — no blocking DDL on large tables, no destructive change without a deprecation period — is worth more than review.