Discord has described moving trillions of message rows from Cassandra to ScyllaDB in 2023, cutting its largest cluster from 177 nodes to 72 and putting Rust data services in front of the store to coalesce reads and cap concurrency to hot partitions. Design the migrator - what has to be true for a per-partition cutover to be safe?

The sequence

1. Dual-write first, read only from the old store. Every write goes to both; the new store is invisible to users. This is the step that makes every later step reversible, and it must run long enough to cover the slowest write path, not merely the common one.
2. Backfill historical data in partition order, idempotently. Each unit of work is one partition, written with the same keys and timestamps as the source so that re-running a unit produces the same result as running it once - which is what lets a crashed migrator restart without reconciliation.
3. Verify per partition before trusting it. Compare counts and a checksum over a canonical serialisation, plus a sampled row-level comparison. Record the verified watermark per partition, because that record is what the cutover reads.
4. Cut over reads per partition, not globally: a flag consulted per key, defaulting to the old store, flipped only for verified partitions. The blast radius of a mistake is then one partition, not the product.
5. Stop dual-writing last, after reads have been served from the new store long enough to see a full weekly cycle including batch jobs and analytics.
6. Decommission with evidence: no reads for a defined quiet period, confirmed from the old store's own request logs rather than from belief.

Where data can diverge, and how you would know

• Late writes during a partition's backfill. If a row is written to the source after the migrator read it but before the dual-write began covering it, it exists in one store only. The protection is ordering: dual-write must start before the backfill, never after.
• Clock and ordering differences. In a last-write-wins store, copying a row with a new timestamp lets a stale value win later. Write the original timestamp, or the migration quietly resurrects deleted or superseded data.
• Deletes. A tombstone that is not copied re-animates a deleted message. Deletes are the most commonly missed case in every migration of this shape.
• How you know: a per-partition diff rate, published as a metric, with a non-zero rate blocking cutover for that partition. A single global "migration complete" boolean has no useful failure mode.

The point of no return

Not the read cutover - stopping the dual write. Until then, reverting is a flag flip. After it, the new store holds writes the old one never saw, and going back requires a reverse migration. The rollback window therefore has a cost: dual-write capacity, double storage, and the engineering attention to keep both correct. Decide the closing date deliberately and write down what evidence closes it, or the overlap becomes permanent.

How long it really takes

The bulk data movement with Discord's purpose-built Rust migrator is reported as about nine days - and the project around it spanned far longer, because the time is dominated by verification, the long tail of odd access patterns, and the dual-write period rather than by copying bytes. Plan the copy in days and the programme in quarters.

When not to do it this way

For a dataset that fits in a maintenance window and a product that can accept one, a stop-the-world copy is cheaper, simpler and safer than a dual-write migration - no divergence, no reconciliation, no migrator to write. The per-partition machinery earns its place when the dataset cannot be copied inside any acceptable window, which is a statement about volume and availability requirements rather than about sophistication.