intermediate 3 min answer

Review this migration plan. Cutover is staged by customer cohort at 2 percent then 10 then 50 then 100. Readiness for each step is "all automated tests green and no open P1 defects". The rehearsal ran against a 5 percent copy of production. The rollback step reads "restore from the pre-cutover snapshot if data integrity is compromised". What would you remove, what would you change, and what would you leave alone?

migration-riskcutoverabort-criteriarehearsalrollback
Show the full answer Hide the answer

What is actually required

A staged cutover needs three things the plan does not have: a numeric condition that stops it, a person named in advance who applies that condition, and a rehearsal that measured duration. Everything else here is reasonable. The plan is not over-built; it is under-decided, and the decisions it defers all land at 03:00 on the people with the most sunk cost in proceeding.

"All tests green and no open P1s" is a pre-condition, not an abort criterion. It is evaluated before the step and says nothing about what to do when the step itself produces a signal nobody predicted. In practice the room improvises, and improvisation at 03:00 by people who have been awake for nineteen hours reliably chooses to continue.

What I would change first

Replace the readiness wording with thresholds that can be read off a dashboard:

  • Abort if the reconciliation mismatch rate across 10,000 sampled records exceeds 0.02%, which at that sample size is roughly two records and is detectable, versus "data integrity is compromised" which is not a measurement.
  • Abort if any step runs past twice its rehearsed duration. Overrun is the earliest honest signal that the plan's model of the system is wrong.
  • Name the person who calls it, and name their deputy. Not a committee. A committee at 03:00 continues.

Then fix the rehearsal. A 5% copy measures correctness and not duration, and duration is the only thing a rehearsal is for. Backfill, index build, replication catch-up and verification are all superlinear in volume or dominated by fixed I/O, so a 5% rehearsal that takes 40 minutes tells you almost nothing about whether the real one takes 13 hours or 30. Rehearse at full volume on a restored copy, and publish each step's measured duration as the number the criteria compare against.

What I would remove

"Restore from the pre-cutover snapshot" as the rollback. Once the 2% cohort has written to the new store, a snapshot restore discards their writes, so it is not a rollback: it is a data-loss event with a procedure attached. Remove the line and replace it with a mechanism — reverse replication from new back to old, kept running through every stage, so rollback for any cohort is a routing change. If reverse replication is not affordable, then say plainly that the point of no return is the first write by the first cohort, and put a name against authorising it.

What I would leave alone

The cohort ramp, including the 2% first step. It looks timid and it is the single most valuable control in the plan: it bounds blast radius to a population you can contact individually, and it exposes the production-only behaviour no rehearsal reproduces. Leave the ramp's shape alone rather than adding stages, because more stages mean more coexistence and coexistence has its own failure modes.

How I would argue this in the review

Not as a list of objections. One question: "at 03:14, reconciliation shows 0.4%, step four is at 2.5 times its rehearsed time, and the business sponsor is asleep. Who decides, against what number?" If nobody in the room can answer, the plan has no abort criteria regardless of what the document says, and that is the finding. The rehearsal and the rollback arguments follow from it rather than competing with it.

When this critique is wrong

If the switch is a routing flag with reverse replication running throughout, there is no partially irreversible step and threshold ceremony slows down the correct behaviour, which is to flip back on any anomaly and investigate afterwards. The same applies to a cohort small enough to revert in minutes. Abort criteria earn their cost where a step is long, partly irreversible, and run by tired people when nobody senior is reachable; where any of those three is false, put the effort into the rollback mechanism and leave the plan's wording alone.