Your DR plan is pilot light with a 30-minute RTO. What would you test, and what will the test probably reveal?
Show the full answer Hide the answer
What the interviewer is testing
Whether you know what actually goes wrong in a failover, most of which is invisible until it is attempted.
What to test
A full failover with a timer running, not a tabletop exercise, and with realistic load rather than synthetic traffic.
What the test will probably reveal
Provisioning takes longer than estimated. Pilot light means most capacity is created at failover time, which depends on the control plane — and control planes are slower under load and may be degraded by the same event. Twenty minutes of provisioning against a 30-minute objective leaves nothing for anything else.
Capacity is unavailable. Everyone else failing over to the same region requests the same instance types simultaneously. Without reservations, capacity is not guaranteed.
Quotas. The secondary region's account has default service quotas sized for the pilot light, not for full production. This is an extremely common failure and it surfaces as a hard stop with a support ticket.
Data replication lag exceeds the stated RPO, or replication has silently been broken for weeks because nothing monitored it.
Dependencies that are not replicated: secrets, certificates, DNS configuration, third-party allowlists keyed to the primary region's addresses, licence servers, identity provider configuration.
Runbooks that are out of date and reference systems that have changed.
The failback path, which is usually undocumented and is harder than the failover.
What to change
Move to warm standby so the failover is a traffic change rather than a build operation — that is the point at which the design becomes statically stable and the recovery time becomes predictable. Reserve capacity in the secondary. Raise quotas in advance. Monitor replication lag with alerting. Test on a schedule, and rotate who runs it so the capability is not held by one person.
What a strong answer adds
Testing failback too. Organisations rehearse failover and then discover during a real event that returning to the primary requires re-replicating data in the other direction, which nobody has scripted.
Common weak answers
A tabletop walkthrough. Testing with the team that built it, at a convenient time, with no load.