advanced 3 min answer

Your cutover plan says switch reads, then switch writes, then decommission. A reviewer asks what happens at 02:00 if the new store turns out to be wrong after writes have moved. What mechanism makes the write switch reversible, and what is the honest unavailability window?

youtubevitesscutoverreverse-replicationrollback
Show the full answer Hide the answer

The asymmetry nobody plans for

Switching reads is a routing change: if the new path is wrong, you route back and nothing was lost. Switching writes is not symmetric, because from that moment the new store holds state the old one has never seen. Rolling back then means restoring from a backup taken before the switch and losing everything written since, which is why teams that have not solved this quietly decide never to roll back.

The mechanism

The fix is to keep replication running in the opposite direction after the switch. At the moment writes move to the new store, a stream starts that applies the new store's changes back onto the old one. The old system stays a warm, current copy rather than a stale snapshot, so rollback becomes the same routing change as it was for reads.

Vitess documents this directly. Its MoveTables workflow takes --enable-reverse-replication when traffic is switched, which sets up replication back to the original keyspace specifically to support rolling the cutover back, and exposes a ReverseTraffic command to do it. Vitess has been the sharding layer under YouTube's MySQL estate since 2011 and graduated in the Cloud Native Computing Foundation in 2019, so this is a mechanism with a long production history rather than a proposal.

The honest unavailability window

It is not zero. To switch writes without losing or duplicating a change, the source has to stop accepting writes, the target has to catch up to the source's last committed position, and only then does routing move. Vitess makes the source read-only for that interval and gates the switch on replication lag with a timeout, so that the attempt aborts instead of extending the freeze.

The number to quote in a review is therefore seconds of write-unavailability, conditional on lag being low at the moment of the switch — typically a handful of seconds when the stream is caught up, and unbounded if you switch while it is behind. That is why the lag gate matters more than the runbook: it converts "we hope it is caught up" into a precondition the tooling enforces.

Where the point of no return actually sits

Not at the write switch. It sits at the moment the reverse stream is stopped, because from then on the old system diverges. Write that step down as its own gated decision with a named authoriser and a minimum soak period after it, and the migration has a reversible phase measured in days rather than a moment measured in seconds.

The cost is real: for the length of the soak you are running two stores, paying for both, and carrying a replication stream whose failure is now a rollback-capability incident rather than a data incident. Alert on the reverse stream's lag exactly as you would on the forward one.

When this is the wrong answer

If the data is small enough that a full restore takes minutes and the business can absorb losing the writes from one quiet hour, a backup and a rehearsed restore is cheaper than a bidirectional stream and has far fewer moving parts. Reverse replication earns its complexity when the restore time or the write volume during the soak is large enough that "roll back" would otherwise mean "do not roll back".