advanced 3 min answer

A programme has one 6-hour maintenance window to cut a payments platform over to its replacement. How many rehearsals should be budgeted and how long will the real cutover take compared with the last rehearsal?

estimationcutoverrehearsalrunbookgo-live
Show the full answer Hide the answer

The assumptions, stated

A cutover runbook with on the order of 60 to 120 discrete steps, several teams, one irreversible step, and a production-like environment available for rehearsal. The quantity being estimated is not the happy path - it is the number of attempts needed before the runbook stops producing surprises.

The arithmetic

Rehearsals find defects in distinct classes, and the classes are found in a predictable order:

  • Rehearsal 1 finds the environment. Missing credentials, a firewall rule, a certificate, a service account, a DNS entry nobody owns. It typically does not complete, and that is the expected outcome rather than a failure.
  • Rehearsal 2 finds the runbook. Steps in the wrong order, two teams waiting on each other, a step that assumed a state the previous one did not leave. It completes slowly, usually over the window.
  • Rehearsal 3 is the first that completes cleanly, and it produces the first trustworthy timing.
  • Rehearsal 4 is run against a changed codebase, because weeks have passed and the system has moved. It validates the procedure under drift, which is a different test from validating it at all.

So three to four rehearsals is the honest budget for a cutover of this size, and fewer than two is a plan to discover the environment class of defect in production.

For duration: a clean rehearsal is run by the people who wrote the runbook, on a system with no live traffic, no customer-facing checks and no decision points. Plan for the real cutover to take 1.5x to 2x the last clean rehearsal, because verification steps are slower when the answer matters, go/no-go decisions take minutes of discussion, and at least one step will be re-run. A rehearsal that completes in 3.5 hours fits a 6-hour window; one that completes in 4.5 does not.

Which assumption dominates the error

How production-like the rehearsal environment is. Rehearsing against a quarter-size dataset invalidates every timing, because the longest steps - backfill, index build, verification scan - scale with data volume and are usually 60–80% of the elapsed time. If the rehearsal data is a tenth of production, the rehearsal measures the decision-making and nothing else.

Second is drift: if the last rehearsal was six weeks before go-live, the runbook has aged against a changing system and the confidence it bought has partly decayed.

What the number rules in or out

  • It rules out a single-rehearsal plan, and it rules out the common compression where "we will rehearse in the window before the real one".
  • It rules in a dry run of the irreversible step against a copy - the one step that cannot be practised in production must be practised somewhere.
  • It rules in a documented abort. If the real cutover is 2x the rehearsal, the abort decision point sits at roughly half the window, and it must be a time, a named decider and a pre-agreed state to return to, rather than a judgement call at hour five. The most expensive cutover is the one that neither completes nor aborts.

Common weak answers

  • "Rehearse until it works." Rehearsals cost environment time and people; four is a budget, and the signal to stop is two consecutive clean runs rather than confidence.
  • "We have automated it, so one dry run is enough." Automation removes the step-execution class of defect and none of the environment or drift classes.
  • "Extend the window if we run over." Maintenance windows are agreed with customers, regulators and partners. The window is a constraint, which is exactly why the 2x factor has to be in the plan rather than discovered inside it.