A programme has one 6-hour maintenance window to cut a payments platform over to its replacement. How many rehearsals should be budgeted and how long will the real cutover take compared with the last rehearsal?
Show the full answer Hide the answer
The assumptions, stated
A cutover runbook with on the order of 60 to 120 discrete steps, several teams, one irreversible step, and a production-like environment available for rehearsal. The quantity being estimated is not the happy path - it is the number of attempts needed before the runbook stops producing surprises.
The arithmetic
Rehearsals find defects in distinct classes, and the classes are found in a predictable order:
- Rehearsal 1 finds the environment. Missing credentials, a firewall rule, a certificate, a service account, a DNS entry nobody owns. It typically does not complete, and that is the expected outcome rather than a failure.
- Rehearsal 2 finds the runbook. Steps in the wrong order, two teams waiting on each other, a step that assumed a state the previous one did not leave. It completes slowly, usually over the window.
- Rehearsal 3 is the first that completes cleanly, and it produces the first trustworthy timing.
- Rehearsal 4 is run against a changed codebase, because weeks have passed and the system has moved. It validates the procedure under drift, which is a different test from validating it at all.
So three to four rehearsals is the honest budget for a cutover of this size, and fewer than two is a plan to discover the environment class of defect in production.
For duration: a clean rehearsal is run by the people who wrote the runbook, on a system with no live traffic, no customer-facing checks and no decision points. Plan for the real cutover to take 1.5x to 2x the last clean rehearsal, because verification steps are slower when the answer matters, go/no-go decisions take minutes of discussion, and at least one step will be re-run. A rehearsal that completes in 3.5 hours fits a 6-hour window; one that completes in 4.5 does not.
Which assumption dominates the error
How production-like the rehearsal environment is. Rehearsing against a quarter-size dataset invalidates every timing, because the longest steps - backfill, index build, verification scan - scale with data volume and are usually 60–80% of the elapsed time. If the rehearsal data is a tenth of production, the rehearsal measures the decision-making and nothing else.
Second is drift: if the last rehearsal was six weeks before go-live, the runbook has aged against a changing system and the confidence it bought has partly decayed.
What the number rules in or out
- It rules out a single-rehearsal plan, and it rules out the common compression where "we will rehearse in the window before the real one".
- It rules in a dry run of the irreversible step against a copy - the one step that cannot be practised in production must be practised somewhere.
- It rules in a documented abort. If the real cutover is 2x the rehearsal, the abort decision point sits at roughly half the window, and it must be a time, a named decider and a pre-agreed state to return to, rather than a judgement call at hour five. The most expensive cutover is the one that neither completes nor aborts.
Common weak answers
- "Rehearse until it works." Rehearsals cost environment time and people; four is a budget, and the signal to stop is two consecutive clean runs rather than confidence.
- "We have automated it, so one dry run is enough." Automation removes the step-execution class of defect and none of the environment or drift classes.
- "Extend the window if we run over." Maintenance windows are agreed with customers, regulators and partners. The window is a constraint, which is exactly why the 2x factor has to be in the plan rather than discovered inside it.