A replacement order system went live two weeks ago and the legacy platform has been read-only since. A defect is found - the new system has been applying the wrong tax rounding to one country since go-live. What happens if you try to roll back?
Show the full answer Hide the answer
Second by second, what rollback means now
The rollback button no longer exists, and the reason is data rather than code. Two weeks of orders live only in the new system. Reverting traffic to the legacy platform gives a system whose state stopped fourteen days ago: those orders cannot be fulfilled, their payments do not reconcile, and customers who can see them in their account history would find them gone.
So the real options are not "roll back or go forward". They are:
- Fix forward and repair the affected rows, which is almost always the answer at this point.
- Roll back and reverse-migrate the two weeks of new data into the legacy schema, which is a migration project executed under incident pressure with no rehearsal.
- Run the legacy platform for the affected country only, if the data partitions cleanly by country - a coexistence decision, not a rollback.
Where it amplifies
Rounding defects compound in the directions that are hardest to undo. Downstream systems have already consumed the wrong numbers: the general ledger has postings, invoices have been issued, a tax return may have been filed, and refunds have been calculated from incorrect totals. Repairing the orders is the small part; repairing their consequences is the project, and some of those consequences involve telling a regulator or a customer.
What the user sees
Nothing, until the correction. Which is the uncomfortable part of this class of defect: a wrong rounding rule produces plausible numbers, so detection depends on someone comparing against an independent calculation rather than on an error appearing anywhere.
What stops it
A rollback window that is kept open deliberately, with a stated price and a closing date. Concretely:
- Reverse sync during the overlap. Writes accepted by the new system are translated back into the legacy schema for an agreed period, so the legacy platform stays a viable destination. It costs real engineering - a second mapping, in the harder direction - and it is the only thing that makes rollback mean anything after go-live.
- A comparison harness that keeps running after cutover, recalculating a sample of the new system's outputs with independent logic. A defect that produces plausible numbers is found by comparison or not at all.
- Partition the cutover by country, tenant or region, so the first weeks of exposure are a few per cent of volume. This is the single cheapest risk reduction available and it is routinely skipped to meet a date.
- A written repair procedure before go-live: how to recompute a field across a date range, who approves it, how downstream consumers are notified.
When reverse sync is the wrong answer
Reverse sync keeps the window open and it is not free: a second mapping in the harder direction, dual capacity, and attention on a system you are trying to retire. Choose it when the cost of being wrong is regulatory or financial - a ledger, a tax calculation, a payments platform - and skip it when the data can be repaired forward, which is most content and workflow systems. It flips as soon as the repair procedure becomes less credible than the reverse migration.
What would have to be true for the window to stay open at all: that no state diverges, or that divergence is reversible by construction - reverse sync running, the legacy platform still processing, and a tested replay procedure. The window closes the moment either system holds data the other cannot reproduce, and in practice that is hours after go-live, not 14 days. A partitioned cutover is the cheap substitute: going live for one country at 3% of volume for 2 weeks bounds the exposure without any reverse mapping at all. Which is why the useful question before any cutover is not "can we roll back?" but "for how long, at what cost, and who decides when it shuts?"