A bucket holds 80 TB and receives 500 GB of new objects a day. Cross-region replication is switched on for disaster recovery and the runbook states an RPO of 15 minutes. What does the arithmetic say about that claim, and what has to change before it is true?
Show the full answer Hide the answer
The assumptions, stated
80 TB existing, 500 GB a day of new objects - about 5.8 MB/s averaged, and realistically a daytime peak several times that. Replication between regions is asynchronous, and the provider replicates objects written after the rule is enabled unless an explicit batch job is run over what is already there.
The arithmetic
The steady-state stream is trivial. 500 GB a day is single-digit megabytes per second; any inter-region path absorbs it, and in normal operation replication lag is seconds to a few minutes. The 15-minute figure is plausible for new objects.
The backfill is the problem.
existing data = 80 TB
sustained backfill ≈ 200 MB/s (a reasonable parallel batch-copy rate)
time = 80,000,000 MB / 200 MB/s ≈ 400,000 s ≈ 4.6 days
So for most of a week after switch-on, the destination holds a fraction of the data - and the runbook's 15 minutes is a statement about new writes that says nothing at all about the 80 TB a recovery would actually need. This is the single most common misunderstanding about replication as a disaster-recovery control.
The second number: catch-up after a burst
If an ingest job writes 2 TB in an hour, the stream is suddenly 20 times the normal rate. Replication queues and drains at whatever throughput the path gives:
backlog = 2 TB at 200 MB/s ≈ 2.8 hours to clear
During those hours the real RPO is hours, not minutes, and nothing in the configuration announced the change. The only way to know is the provider's replication-latency and pending-bytes metrics, which is what the runbook should reference instead of a constant.
Which assumption dominates the error
The achievable parallel copy rate, which varies by an order of magnitude with object size: 80 TB in 50 MB objects copies quickly, and 80 TB in 20 KB objects is 4 billion requests and is bounded by request rate rather than bandwidth. Compute the object count before promising a backfill date.
What else breaks the DR claim
- Deletes. Delete markers and versioned deletions are not necessarily replicated by default, so the destination keeps objects the source no longer has. For recovery that is usually harmless and for compliance deletion it is a finding.
- Replication is not a point in time. It copies corruption and accidental overwrites as faithfully as it copies good data, so it is not a substitute for versioning or backups.
- Metadata and storage class may not carry over as assumed, and a destination in a cheaper class can have retrieval times that break the recovery you planned.
What to change before the claim is true
- Run the batch replication job over existing objects and record the completion date.
- Replace the constant in the runbook with a measured metric - replication latency and pending bytes, alerted on - because the real RPO is whatever that metric says during an incident.
- Replicate deletes explicitly if the destination is also the compliance copy, or document that it is not.
- Keep versioning and a separate backup, because replication covers region loss and nothing else.
Why the other options fail
- "Replication is synchronous." No major object store replicates across regions synchronously - the latency would be charged to every write. Asynchronous is the design, and asynchronous means a window.
- "RPO depends on object size so small objects are faster." Small objects are worse, because the constraint becomes requests per second rather than bytes per second.
- "It re-copies the bucket automatically." The common default is that only new writes are replicated. Believing otherwise is how a destination bucket ends up holding three weeks of a five-year history.
When this is the wrong control entirely
If the requirement is "we must not lose data to a mistake", replication is the wrong purchase at any RPO, because it copies the mistake. Spend the money on versioning with a lifecycle, and on an immutable copy in a separate account, and treat cross-region replication as a latency and region-loss control rather than a data-protection one. For a bucket whose contents can be regenerated from a system of record, the honest answer may be to replicate nothing and re-derive on recovery.