Rollback Window

Every cutover plan has a rollback section and most of them are obsolete within hours of go-live. The code can be reverted at any time; the data cannot. Once the new system has accepted writes that the legacy platform never saw, reverting traffic produces a system whose state stopped at go-live: yesterday's orders are unfulfillable, yesterday's payments do not reconcile, and customers see history vanish.

The rollback window is that period stated honestly: from the change until the first state divergence that cannot be reproduced on the other side. For a stateless deployment it is indefinite. For a system of record it is usually hours, and the plan that assumes otherwise is the risk.

Why it matters

Because the defects that force a rollback are frequently found long after the window has closed. A wrong tax rounding, a mis-mapped entitlement or a broken fee calculation produces plausible output, so it is detected by a month-end reconciliation or a customer complaint rather than by an alert. By then the only options are repair-forward, a reverse migration under pressure, or running the legacy system for the affected slice.

Naming the window also changes the plan for the better. A team that knows it has four hours builds a partitioned cutover and a repair procedure; a team that believes it can roll back for a month builds neither.

Implementation patterns

• State the window in the plan with its closing condition, not as a promise of reversibility. "Rollback is available until dual-write stops, scheduled for day 14" is a commitment; "we can roll back" is not.
• Keep it open deliberately with reverse sync where the data is financial or regulated, accepting its cost: a second mapping, dual capacity, and attention on a system being retired.
• Shrink what is at risk instead of extending the window. A partitioned cutover - one country, one tenant, one hash range at a few per cent of volume - is the cheapest risk reduction available and is routinely dropped to meet a date.
• Write the repair procedure before go-live: how to recompute a field over a date range, who approves it, how downstream consumers are told. After the window closes, repair-forward is the plan, so it should be a rehearsed one.
• Monitor reverse-sync lag as the data-loss figure of any rollback taken now.
• Set an abort point inside the cutover itself - a time, a named decider, a defined state to return to - because the most expensive cutover is one that neither completes nor aborts.

Industry example

Incremental migrations are the industry's answer to a short window. Slack's move to Vitess proceeded keyspace by keyspace over roughly three years, and Notion's 2021 sharding work moved into 480 logical shards in stages rather than in one event; in both cases the reversible unit was a slice rather than the product. The contrast case is instructive: Knight Capital's 2012 incident, as the SEC's 2013 order describes, lost roughly $460 million in about 45 minutes - damage can outrun any deployment-level rollback, which is why a kill switch outside the affected path matters more than the ability to redeploy.

Failure scenarios

• A rollback plan that assumes code reversion is enough, discovered during the incident.
• Reverse sync that was never exercised, failing on first use.
• An overlap period with no end date, so dual-write becomes permanent and the migration never finishes.
• A defect found after the window whose consequences have already propagated to the ledger, to issued invoices, and possibly to a filed return - where repairing the records is the small part.
• Partitioned cutover skipped to hit a date, converting a 3% exposure into a 100% one.
• The window closed silently, because nobody was tracking which system held unreproducible state.

Trade-offs

Keeping the window open costs real money and attention: dual capacity, a reverse mapping, two systems to keep correct, and a delayed decommission that defers the saving the programme was funded for. Closing it early banks the saving and transfers the risk to the repair procedure. The useful framing is not rollback versus no rollback, but which of three instruments to buy - a longer window, a smaller exposure, or a better repair capability. For most products, exposure reduction is the cheapest of the three; for ledgers and regulated calculations, the window is worth its price.

When not to use it

For stateless deployments this concept does not bind - revert the artefact and the question does not arise - and treating every release as if it had a closing window produces ceremony without benefit. It is also the wrong focus where the real protection is forward-only by design: schema changes made with expand-contract, feature flags that gate behaviour rather than data shape, and append-only stores can be corrected in place, so the engineering should go into the repair path rather than into preserving a route backwards.

Interview question

Q: A team tells you the rollback plan for next month's cutover is to redeploy the previous version. What is wrong with that, and what would you ask them to add?

What a strong answer covers: separating code reversion from data divergence and identifying the first unreproducible write as the window's true end; asking for the window to be stated with a closing condition; proposing a partitioned cutover to shrink exposure instead of extending the window; requiring a written and rehearsed repair procedure, since repair-forward will be the real plan; reverse sync only where the data justifies it, with lag monitored as the rollback's data-loss figure; and an in-cutover abort point with a time and a named decider.

Quick check

Quiz: What closes a rollback window? The first write of state that the other system cannot reproduce - typically hours after go-live for a system of record.

Flashcard: Cheapest way to reduce cutover risk when the window cannot be extended? Partition the cutover - go live for one tenant, country or hash range at a few per cent of volume - so the exposure is bounded without any reverse mapping.