A platform must migrate very large volumes of file data between storage systems. Which strategies apply, and what dominates the schedule?
Show the full answer Hide the answer
The strategies
1. Bulk copy then incremental catch-up. Copy everything, then replay changes since the copy started, then cut over. Simple, and the catch-up phase must converge — if the change rate exceeds the catch-up rate, it never completes.
2. Lazy migration on access. Move an object when it is next requested, with a background sweep for the tail. Excellent when access is skewed, which it usually is: the hot minority migrates quickly and naturally, and the cold majority migrates in the background without competing for the same capacity.
3. Dual-write with backfill. New writes go to both; history backfills. Keeps both current and requires reconciliation.
4. Read-through migration. Reads check the new store and fall back to the old, migrating on the way. Combines lazy migration with a clean read path.
For very large stores with skewed access, lazy migration on access plus a background sweep is usually the right shape, because it aligns migration effort with actual usage.
What dominates the schedule
Not the copy — the verification and the tail.
Copying data at scale is a bandwidth and cost problem with a calculable duration. What extends the schedule:
- Verification. Proving that every object arrived intact, at scale, requires reading everything again. Content hashing makes this tractable, and it is not free.
- The long tail. The last small percentage — objects with unusual metadata, corrupted records, objects referenced by nothing, objects that fail to copy for individual reasons — takes disproportionately long and requires case-by-case handling.
- Deduplication and reference integrity. In a content-addressed store, moving blocks while preserving reference counts across users is delicate, and getting it wrong deletes one user's data when another deletes theirs.
- Cutover coordination, especially where clients cache locations.
The properties that make it safe
- Content addressing and hash verification, so integrity is provable rather than assumed.
- Idempotent transfer, so a restart does not duplicate or corrupt.
- Per-tenant or per-shard cutover, bounding the blast radius of any mistake.
- The source retained and restorable until verification is complete and a retention period has elapsed.
- Reconciliation for the deduplicated case, since reference counts diverging is a silent data-loss risk.
The rule
Never delete the source until verification has passed and a retention window has elapsed. The cost of retaining data longer is small and calculable; the cost of discovering a verification gap after deletion is unbounded.