A DR test fails: the secondary region cannot launch enough instances. What happened, and what standing checks prevent it?

What happened

Service quotas in the secondary region are far lower than in the primary, because nothing has ever run there at scale. Quotas are per account and per region, and the primary's limits were raised incrementally over years as the workload grew — each increase a request nobody recorded as a dependency.

The failover attempts to launch capacity and is refused. The autoscaler is working correctly; the cloud is declining.

This is the most common pilot-light and DR failure, and it is entirely preventable.

The standing checks

1. Inventory the quotas that matter per critical workload — instance counts by family, IP addresses, load balancers, function concurrency, database instances, storage volumes, API rate limits — and record current usage against each.

2. Assert parity between primary and secondary regions, automated and reported. This should be a standing item in the DR review rather than something discovered during a test.

3. Alert on utilisation at 70–80%, not on the ceiling. Discovering a limit by reaching it during a spike is the other common failure.

4. Raise limits ahead of known events. Quota increases are requests with lead time — sometimes days — not settings that take effect immediately.

5. Test the failover at realistic scale, not with a token workload. A DR test that launches five instances proves nothing about launching five hundred.

The related exposures worth checking at the same time

Capacity availability, not just quota. In a genuine regional event, everyone fails over simultaneously and the secondary region may not have the physical capacity even if your quota permits it. Capacity reservations exist for exactly this and are the honest answer for critical workloads.

Provisioning depends on the control plane, which may itself be degraded during a regional incident — which is the argument for warm standby over pilot light where RTO genuinely matters.

The governance point

Quotas cut both ways. They are a failure mode when too low and a guardrail when set deliberately — a low concurrency limit on a non-production account is what stops a runaway loop producing a very large bill. They should be a managed property with an owner, not a default nobody has examined.

What a strong answer adds

Noting that the DR test succeeded in the most important sense: it found a defect that would otherwise have surfaced during a real regional failure. A DR test that always passes is usually testing too little, and finding this is the return on running it.