An auditor requires evidence that a four-hour RTO is achievable for a system that has never failed over. Production cannot be risked and the business will not accept an unplanned outage. Sequence the first real disaster-recovery test.

The sequence, each step reversible

1. Write down what "recovered" means, before anything is scheduled. A named business transaction succeeding end to end from outside the network, at a stated rate, with data no older than the stated RPO. Without this, the test produces an argument about what counts rather than evidence. It also usually reveals that the documented RTO was never defined against a user-visible outcome.

2. Inventory dependencies the plan does not mention. Identity provider, DNS, certificates, secrets, third-party APIs with source-address allowlists, licence servers, and any data store outside the primary database. The standard finding of a first DR test is that something essential exists in one region only, and it is cheaper to find that on paper.

3. Restore to an isolated environment and time it. Nothing in production is touched. This establishes the restore time and, more importantly, whether the restore works at all. Reversible by deleting the environment.

4. Run the recovered stack against production-shaped synthetic traffic, still isolated. This finds the capacity problem: standby environments are routinely scaled down, and a recovery that cannot take peak load is not a recovery. Reversible.

5. Dark failover: bring the secondary to full readiness and serve internal or synthetic traffic only, with production untouched. This exercises promotion, credentials and connectivity. Reversible by leaving the secondary running or tearing it down.

6. A scoped live failover, during a low-traffic window, with a fixed abort time. One region, announced, with the rollback decision pre-agreed and delegated to the person running the test rather than to a committee. The abort criterion is a time and a metric written before the test, because judgement at minute 40 of a failing exercise is poor.

7. Fail back deliberately, and measure it. Fail-back is usually harder than failover, is almost never rehearsed, and is where data written during the exercise has to be reconciled.

Where data can diverge, and how you would know

Between the last replicated write and the promotion, and again during fail-back if both sides took writes. Before step 6, decide whether the secondary will accept writes at all; a read-only failover is a far weaker test and a far safer one, and it is a legitimate first step. Detect divergence by comparing row counts and checksums for the highest-value tables, and by reconciling a business-level total such as the day's transaction sum.

The point of no return

Promotion of the secondary database to primary. Everything before it is a rehearsal; after it, returning means a fail-back with reconciliation. Know which step you are on, and say it out loud during the test, because the most common failure of a first DR exercise is people not realising they have crossed it.

How long it really takes

Steps 1 to 4 take weeks, mostly spent discovering undocumented dependencies. The first live exercise finds three to five things nobody knew, and the measured RTO on a first attempt is routinely two to five times the documented figure. The honest output of the first test is a real number, and it is usually worse than the promise - which is exactly the evidence the auditor asked for, and better delivered by you than by an incident.

When not to go all the way

If the business cannot accept even a scoped live failover, stop at step 5 and document that. A rehearsed dark failover with measured restore and capacity evidence is worth far more than an argument about whether to do a live one, and it is a credible intermediate position with an auditor, provided the gap is stated rather than hidden.