advanced 2 min answer

A platform must migrate very large volumes of file data between storage systems. Which strategies apply, and what dominates the schedule?

data-migrationbulk-transferverificationcutoverdropboxdesign
Show the full answer Hide the answer

The strategies

1. Bulk copy then incremental catch-up. Copy everything, then replay changes since the copy started, then cut over. Simple, and the catch-up phase must converge — if the change rate exceeds the catch-up rate, it never completes.

2. Lazy migration on access. Move an object when it is next requested, with a background sweep for the tail. Excellent when access is skewed, which it usually is: the hot minority migrates quickly and naturally, and the cold majority migrates in the background without competing for the same capacity.

3. Dual-write with backfill. New writes go to both; history backfills. Keeps both current and requires reconciliation.

4. Read-through migration. Reads check the new store and fall back to the old, migrating on the way. Combines lazy migration with a clean read path.

For very large stores with skewed access, lazy migration on access plus a background sweep is usually the right shape, because it aligns migration effort with actual usage.

What dominates the schedule

Not the copy — the verification and the tail.

Copying data at scale is a bandwidth and cost problem with a calculable duration. What extends the schedule:

  • Verification. Proving that every object arrived intact, at scale, requires reading everything again. Content hashing makes this tractable, and it is not free.
  • The long tail. The last small percentage — objects with unusual metadata, corrupted records, objects referenced by nothing, objects that fail to copy for individual reasons — takes disproportionately long and requires case-by-case handling.
  • Deduplication and reference integrity. In a content-addressed store, moving blocks while preserving reference counts across users is delicate, and getting it wrong deletes one user's data when another deletes theirs.
  • Cutover coordination, especially where clients cache locations.

The properties that make it safe

  • Content addressing and hash verification, so integrity is provable rather than assumed.
  • Idempotent transfer, so a restart does not duplicate or corrupt.
  • Per-tenant or per-shard cutover, bounding the blast radius of any mistake.
  • The source retained and restorable until verification is complete and a retention period has elapsed.
  • Reconciliation for the deduplicated case, since reference counts diverging is a silent data-loss risk.

The rule

Never delete the source until verification has passed and a retention window has elapsed. The cost of retaining data longer is small and calculable; the cost of discovering a verification gap after deletion is unbounded.