advanced 3 min answer

Interview prompt. Your recovery point objective is 15 minutes. I am an auditor and I do not accept your replication dashboard. Prove it.

rpobackupsrestoreverificationcorruption
Show the full answer Hide the answer

What the interviewer is testing

Whether you know the difference between a replication metric and a recovery guarantee. Replication lag tells you how far behind the replica is from the last successfully shipped transaction. It says nothing about whether that replica can be opened, whether the data in it is correct, or whether the last 15 minutes of shipping has been happening at all.

The second thing being tested: whether you can produce evidence rather than a dashboard. An auditor asking for proof is asking for a repeatable procedure with a recorded result.

The clarifying questions that change the answer

  • RPO measured from what event? A clean failover, a hardware loss, a region loss, or a logical corruption. The first three are a replication question; the fourth is not, and it is the one that actually loses data.
  • Is replication synchronous or asynchronous? Asynchronous means the RPO is bounded by lag, which is a distribution rather than a number, so the claim is really about its 99th percentile.
  • What is the slowest-shipping component? The RPO of a system is the worst of its parts: the database, the object store, the search index, the message broker, the configuration store. A 15-minute database RPO plus a 24-hour object-store backup is a 24-hour RPO for anything that needs both.
  • Who verifies the backups? Not "do they run" but "has one been restored, opened, and compared".

The evidence a strong answer offers

1. A point-in-time restore, performed on a schedule, into an isolated environment, with a recorded diff. Pick a timestamp, restore to it, and compare a sample of rows against what the source held at that moment. The output — the timestamp restored to, the actual data loss measured, the wall-clock time — is the evidence. An RPO that has never been measured by restoring is an aspiration.

2. The measurement repeated for every component in the recovery scope, with the worst one quoted as the system's RPO.

3. The logical-corruption case treated separately. Replication faithfully replicates corruption, and this is the scenario that defeats most RPO claims: a bad migration or a buggy release writes wrong data, and every replica has it within milliseconds. Protection against this is retention of point-in-time recovery for a window longer than your detection time, which means the real question is "how long until we notice?" — and if detection takes 3 days, a 15-minute RPO is irrelevant.

4. Evidence that the shipping has not silently stopped. The GitLab incident of January 2017 is the standing example: several backup and replication mechanisms had not been working and nothing reported it, which was discovered when roughly six hours of data was lost. The control is a positive assertion — the restore test itself — rather than the absence of an alert.

Common weak answers

  • "Our replication lag is under a second." Measures the wrong thing and measures it only while everything works.
  • "Backups run nightly and we get a success email." A success email records that a process exited zero. It does not record that the archive is readable or that it contains what you think.
  • "We have a documented DR plan." A plan is not a measurement. The auditor is asking for a result.
  • "We failed over once in 2023 and it worked." Better than nothing, and it proves RTO rather than RPO, and it proves it for the configuration as it stood then.

What a strong answer adds

The honest number. Most organisations that run this test for the first time find their real RPO is worse than claimed, usually because of one lagging component nobody included in the scope — the object store, the search index, or a configuration service. Saying so, with the measurement and a dated plan to close the gap, is a stronger position than a confident claim with no evidence behind it.

And the second-order point: RPO and detection time multiply in importance. A tight RPO with slow detection protects against infrastructure loss and not against the far more common cause of data loss, which is the system writing the wrong thing correctly.