A marketplace's failover mechanism requires calling a control plane to provision replacement capacity. Why is this a design flaw, and what is the alternative?

The principle

Static stability means a system continues operating correctly during a failure without requiring any change. No scaling action, no configuration push, no deployment, no control-plane call.

The reasoning is a correlation argument: the events that trigger failover — a regional failure, a large demand shock, a network partition — are precisely the events most likely to degrade the control plane, or to have every other affected system calling it simultaneously.

A recovery mechanism that depends on the control plane has taken a dependency on something whose availability is negatively correlated with your need for it. That is the worst possible dependency profile, and it is common because it is invisible in normal operation.

What it looks like in practice

Capacity that already exists. Surviving regions run with enough headroom to absorb a failed peer, rather than planning to scale up when it fails. This is expensive — often 20–30% of the fleet idle for an event that may not happen this year — and it is exactly the cost most often cut in budget review, which is why regional redundancy so frequently does not work when needed.

Configuration already distributed. Every node holds a locally cached last-known-good configuration and continues on it if the control plane is unreachable. It does not need to fetch anything to keep serving.

Failover as a routing change, not a provisioning action. Withdrawing a health signal moves traffic to capacity that is already running.

Defaults that fail safe. If a component cannot reach its configuration source, it continues with what it has rather than stopping or reverting to an empty state.

The general test

For each recovery mechanism ask: what must succeed for this to work, and how likely is that thing to be healthy during the failure it recovers from?

Common answers that should worry you: an autoscaling API, a container registry, a secrets manager, a service discovery system, a deployment pipeline, a DNS provider. Each is fine in isolation and each is a control-plane dependency on the recovery path.

The trade-off, stated honestly

Static stability costs idle capacity and pre-distributed state, which is real money spent continuously against an event that may never occur.

The alternative is a recovery mechanism that works in every test — because tests are run when everything else is healthy — and may not work during the one event it exists for. If the headroom is genuinely unaffordable, the honest response is to reduce the size of the failure domain — smaller cells, more of them — rather than to assume the control plane will be available.