Reverse Replication Cutover

Switching reads to a new system is reversible by construction: if the new path is wrong, route back, and nothing was lost. Switching writes is not. From the moment the first write lands only in the new store, the old system is a stale snapshot, and rolling back means restoring a backup and discarding everything written since.

Teams discover this in the review, not in the design, and the usual response is to promise very hard that rollback will not be needed. That is how a plan with a rollback section on paper becomes a plan with no rollback in practice.

The pattern closes the gap with a mechanism: at the moment writes move, start replication in the opposite direction, so the old system stays a current, warm copy for as long as the new one is on trial.

Why it matters

The cost of a migration failure is not the defect. It is the time between discovering the defect and being back in a known-good state, multiplied by whatever the business loses per hour in that state. Reverse replication collapses that interval from a restore-and-reconcile exercise measured in hours or days to a routing change measured in seconds.

It also changes behaviour before anything goes wrong. A team that genuinely can roll back will cut over earlier, in smaller pieces, with less ceremony, because each step is cheap to undo. A team that cannot will batch changes into one enormous event, which is precisely the shape that fails.

Implementation patterns

• Start the reverse stream as part of the switch, not after it. If it is a follow-up task, there is a window in which rollback is impossible and nobody has noticed.
• Make the source read-only while the target catches up, then move routing. This is what prevents a write landing in the old store after the switch and producing two divergent lineages.
• Gate the switch on replication lag with a timeout. If the stream is behind, the switch aborts rather than extending the read-only window. Enforcing this in tooling converts "we think it is caught up" into a precondition.
• Alert on the reverse stream's lag exactly as on the forward one. Once writes have moved, a stalled reverse stream is a loss of rollback capability, and it is silent.
• Name the stop of the reverse stream as its own gated decision with an authoriser and a minimum soak period before it. That step, not the write switch, is the point after which divergence begins.
• Constrain the new system's value domain during the soak so it cannot write anything the old schema is unable to hold.

Industry example

Vitess implements this directly. Its MoveTables workflow accepts --enable-reverse-replication when traffic is switched, establishing replication back to the original keyspace specifically to support rolling the cutover back, with a ReverseTraffic command to perform it. The documentation is explicit that switching write traffic makes the source read-only while targets catch up, so the honest number is a short window of write-unavailability rather than zero — on the order of 5 to 30 seconds when the stream is caught up — and that the switch can be gated on current lag to bound it.

Vitess has been the sharding layer under YouTube's MySQL estate since 2011 and graduated in the Cloud Native Computing Foundation in 2019.

Failure scenarios

• The reverse stream stops and nobody notices. Rollback capability is gone and the dashboard is green, because no user-facing signal depends on it.
• A write reaches the old store after the switch, from a forgotten batch job or an admin script, and the two lineages diverge. Now neither direction is safe.
• The new system writes a value the old schema cannot represent — a new enum member, a longer field, a nullable column made mandatory — and the reverse stream fails on that row or silently drops it. Reversibility ended at that write, not at the cutover.
• The switch is attempted while lag is high, the read-only window extends from seconds to minutes, and what was a cutover becomes an incident.
• The soak period is skipped because everything looked fine on day one, and the defect that appears in week three has no cheap way back.

Trade-offs

When not to use it

When a rehearsed restore is fast enough and the write volume during the soak is small. If the dataset restores in minutes and the business can absorb losing an hour of quiet-period writes, a backup plus a tested restore procedure is cheaper and has no silent failure mode. Reverse replication earns its complexity only when restore time or write volume would otherwise make "roll back" mean "do not roll back".

It is also the wrong tool where the two systems have genuinely different data models. A stream translating in both directions is two mappings that must stay consistent, and the mapping becomes the thing most likely to fail. There, prefer a shorter soak with a hard freeze and a narrower rollback you have written down.

Interview question

Q: You are moving writes from one datastore to another at 02:00. Your plan says "roll back if anything is wrong". Tell me precisely what that sentence means at 02:15, what it means on the third night, and what you would have had to build during the day for either answer to be true.

What a strong answer covers: that rollback after a write switch is meaningless without a mechanism to keep the old store current · reverse replication as that mechanism, started as part of the switch · the read-only catch-up window and the lag gate that bounds it · that the point of no return is the stopping of the reverse stream and needs an authoriser · that new values outside the old schema's domain end reversibility silently · and the alternative when restore is fast and volume is low.

Quick check

Quiz: After write traffic moves to the new store, what makes a rollback cheap rather than destructive? A replication stream running from the new store back to the old one, so the old system stays current and rollback is a routing change.

Flashcard: Why is switching writes not symmetric with switching reads, and what restores the symmetry? — Because the new store immediately holds state the old one has never seen, so rollback would mean a restore; reverse replication keeps the old store current and makes rollback a routing change again, at the price of running both stores through the soak.