Metastable Failure

A metastable failure has two properties that make it distinctive and dangerous:

1. It is triggered by something small — a brief latency blip, a deploy, a cache flush — that would normally be absorbed.
2. It persists after the trigger is removed. Fixing the original cause does not restore service, because the system is now generating enough self-inflicted load to stay broken.

The system has two stable states — healthy and collapsed — and a shove moves it from one to the other with no path back.

Why it matters

Most incident response assumes causality runs one way: find the cause, fix it, service returns. In a metastable failure, that assumption is false, and the team burns the incident hunting a cause that has already gone away. The only thing that works is shedding load — which feels wrong, because the system looks like it needs more capacity, not less.

How it forms

Almost always a positive feedback loop involving retries or reconnects:

• Latency rises → clients time out → clients retry → load increases → latency rises further.
• A node fails → its connections reconnect elsewhere → the survivors are overloaded → more nodes fail → more reconnects.
• A cache flushes → misses hit the database → the database slows → cache fills slowly → misses persist.

The amplification is multiplicative through the call graph: three retries at each of three layers is 27x, not 3x.

Implementation patterns that prevent it

• Retry budgets capping retries as a fraction of successful requests. When everything is failing, almost nothing is retried — the correct behaviour, and one that per-request retry counts cannot express.
• Full jitter on backoff. Backoff alone moves the synchronised wave; jitter dissolves it.
• Deadline propagation, so work whose caller has already given up is abandoned rather than completed.
• Load shedding at admission, rejecting fast so the system does useful work for a subset rather than useless work for everyone.
• Bounded queues. An unbounded queue is a mechanism for accumulating work that will be discarded.
• Recovery that is not synchronised — staged restarts, gradual traffic return, cache warming before full load.

Industry example

A mobility platform sees a regional dependency slow down — not fail. Callers hit timeouts and retry; each retry adds load; the dependency slows further. The original blip passes within a minute, and the outage continues for an hour. Nothing recovers until traffic is manually cut, at which point the system returns to health almost immediately — the classic signature.

The same shape appears in reconnect storms on persistent-connection platforms, in thundering herds after a cache invalidation, and in database connection-pool exhaustion where each application instance opens more connections in response to slowness.

What all of these share is that every individual actor is behaving rationally. The retry improves that request's chance of success. The reconnect is necessary. The extra connection is reasonable. The aggregate is collapse — which is why the controls must live in shared infrastructure rather than being each team's decision.

Failure scenarios

• Adding capacity during the event, which the new capacity immediately consumes serving retry traffic.
• Rolling restarts, which produce a fresh wave of reconnects and cold caches.
• Removing the trigger and declaring victory, then watching the outage continue.
• Load shedding configured but never exercised, so it fails on first use.

Trade-offs

The controls cost throughput in the normal case: retry budgets mean some transient failures are not retried and surface to users; load shedding means rejecting requests the system could probably have served; bounded queues mean refusing work during brief spikes. You are buying insurance against collapse with a small, continuous premium — and organisations under cost pressure routinely cancel the policy, usually shortly before they need it.

Interview question

"An outage began with a 30-second dependency blip an hour ago. The dependency has been healthy for 55 minutes and your service is still down. What is happening, and what is the first thing you do?"