pattern

Pre-Provisioned Capacity

Running capacity that is already in place to absorb a failure, rather than depending on a control plane to create it during the failure.

The concrete expression of static stability. If losing a zone requires launching instances to recover, then recovery depends on the control plane — the API that provisions, scales and configures — and control planes are more complex, change more often, and are more likely to be degraded during exactly the event that requires them.

The pattern: run enough capacity across zones that losing one leaves the remainder able to serve peak demand with no scaling action at all. The system does not recover; it never degraded.

The same principle applied elsewhere: cache configuration and credentials locally with a long fallback so a configuration service outage is invisible; keep last-known-good service discovery data rather than failing when discovery is unreachable; and fail static — return a defined default — instead of failing in whichever direction the code happens to take.

The cost is real and should be stated plainly: utilisation is deliberately lower, so the infrastructure bill is higher. That is the price of not depending on a dependency during its most likely failure.

The general test for any recovery mechanism: what does this require to be working, and is that thing likely to be working when this fires?