A frequently written table needs a column renamed and its type changed. The service runs many instances and cannot be stopped. How do you do it and remain able to roll back at every step?

What is being tested

Whether you know the expand-and-contract pattern, and — more importantly — whether you understand why each step is separate, which is that during a rolling deployment two versions of the code are running at once.

The core insight

At no point can you assume all instances run the same code. During any rollout, old and new coexist for minutes. Therefore every intermediate state must be readable and writable by both versions. That single constraint dictates the entire sequence.

This is the same discipline that operating at very large scale forces permanently. Meta cannot coordinate an upgrade across billions of deployed mobile clients, so every schema and API change must be compatible with every version already in the wild; additive change and long overlap windows are not a nicety there, they are the only available mechanism.

The sequence

1. Expand — add the new column. Nullable, no default that requires a table rewrite, no constraints. Deploy. Nothing reads it. Rollback: drop the column.

2. Dual-write. Deploy code that writes both old and new columns on every insert and update, but still reads only the old one. Rollback: revert the code; the old column is still authoritative and complete.

3. Backfill. Copy historical rows in batches, with a rate limit, resumable from a cursor, in chunks small enough not to hold long locks. Run it as a job you can pause. Rollback: stop the job; nothing reads the new column yet.

4. Verify. Run a comparison job over a sample — ideally all rows — checking that old and new agree, including for rows written during the backfill. Do not skip this. This is the step that catches the type conversion that silently truncated something.

5. Read from new, behind a flag. Flip a small percentage of traffic to read the new column while still dual-writing. Compare error rates and results. Rollback: flip the flag back — instant, no deployment.

6. Read from new everywhere. Still dual-writing. Rollback is still a flag flip.

7. Stop writing the old column. Now rollback requires a code deployment plus a backfill of the gap, so this is the first genuinely one-way door. Wait long enough here that you are confident.

8. Contract — drop the old column. After a deliberate soak period, measured in weeks rather than hours for anything important.

What people get wrong

• Combining steps 2 and 5 — deploying dual-write and new-read together — which means a rollback during the rollout leaves rows written only to the new column and invisible to the old code.
• Forgetting the other writers. Batch jobs, admin tools, data pipelines and the analytics extract also touch this table. Every one of them needs the same treatment.
• Backfilling in one statement, which locks the table and takes the service down for the duration.
• Dropping the old column the same week. The rollback you need is always the one you already discarded.

The general principle

An additive change is reversible; a destructive change is not. Sequence work so that every irreversible step happens last, alone, and after verification — and so that the number of irreversible steps is as small as you can make it.