500 TB of object data must move to another provider over a 10 Gbps dedicated link while the source keeps receiving 2 TB of new objects a day. Give the sequence, and the arithmetic that decides whether the link is enough.
Show the full answer Hide the answer
The arithmetic first, because it decides the plan
10 Gbps = 1.25 GB/s at line rate
realistic sustained (60-70%) ≈ 0.8 GB/s
per day ≈ 0.8 × 86,400 ≈ 69 TB
500 TB ≈ 7.3 days of continuous transfer
incoming 2 TB/day ≈ 3% of transfer capacity - not the constraint
So the link is sufficient on bytes. Now check the other dimension, which is where this plan usually fails: 500 TB in objects averaging 20 KB is 25 billion objects. At a few hundred object operations per second per worker, that is years of work serially, so the real design problem is parallelism and request rate, not bandwidth. Hundreds of workers partitioned by key prefix, with the provider's request-rate limits as the binding constraint.
The rule to carry away: for large objects the bottleneck is the link; for small objects it is operations per second and metadata. Compute the object count before specifying the network.
The sequence
- Inventory and checksum manifest of the source, so completeness is a comparison rather than a belief.
- Bulk copy in parallel by key prefix, idempotent and restartable, with per-object checksum verification on write.
- Incremental passes over objects created or modified since the manifest, repeated until the delta per pass is small.
- Dual-write new objects to both providers for the final stretch, or pause writes if the business can take a window - dual-write is more work and the only option for a continuously-written store.
- Final catch-up and reconciliation - count, total bytes and a sampled checksum comparison.
- Cut over reads by switching the path, keeping the source authoritative for rollback.
- Keep the source read-only for a rollback window of days to weeks before deleting anything.
Where data can diverge, and how you would know
- Objects modified during the bulk copy, which the incremental passes catch only if they are driven by modification time or a change feed - a copy driven by a static listing misses them silently.
- Deletes, which are the classic omission: an object deleted at the source after being copied stays alive at the destination forever unless deletes are replicated explicitly.
- Metadata: storage class, custom metadata, content type, retention locks and access policies are not carried by a naive byte copy, and their absence is discovered by the application rather than by the copy tool.
The detection is the manifest comparison: counts and bytes catch bulk problems, sampled checksums catch corruption, and an explicit delete-reconciliation pass catches the third.
The point of no return
Deleting source data, or beginning writes to the destination without dual-writing. Everything before that is reversible by pointing reads back.
What about a physical transfer appliance
The decision rule: an appliance holds tens of terabytes up to roughly 100 TB per unit and takes about a week door-to-door, so it wins when the available link is below roughly 1 Gbps, or when the transfer window is shorter than the arithmetic allows. At 10 Gbps and 7 days it loses - and it is strictly worse for 25 billion small objects, which have to be read and written one at a time regardless of how the bytes travel.
The cost nobody budgets
Egress. At the major providers' 2026 list prices, egress runs on the order of tens of dollars per terabyte after volume tiers, so 500 TB is a five-figure line item. Check the exit terms before assuming it: since 2024 the large providers have offered free data transfer for customers migrating off entirely, under an application process and with conditions, following regulatory pressure in Europe. A migration that qualifies pays nothing; one that keeps a foot in both camps pays list.
When this is the wrong approach
If the data is cold and queried rarely, do not move it at all: leave it where it is, pay the storage, and move only the access path. The cheapest migration of a 500 TB archive is often the one you do not perform - and that option disappears the moment someone frames the project as "exit the provider" rather than "move the workloads that matter".