Recovery Load

When a system recovers, the load it faces is not the load it normally serves. Hours of accumulated work is released in minutes, every disconnected client reconnects at once, caches are empty so every read reaches the origin, and queued retries fire simultaneously.

Recovery load is frequently an order of magnitude above steady-state, and it arrives at the moment the system is least able to absorb it — partially restored, with cold caches, possibly with reduced capacity, and operated by people who have been awake for some time.

Why it matters

The characteristic incident shape is a short trigger followed by a long outage caused by the recovery. The original fault was fixed in minutes; the system then spends hours unable to reach a serving state because each recovery attempt reproduces the overload.

Post-incident analysis that focuses on the trigger learns nothing durable, because the trigger will be different next time and the amplification will be identical.

It also means steady-state capacity planning is insufficient by construction. A system sized for a million concurrent connections has no capacity for a million connection establishments per minute, and the gap is often two orders of magnitude.

Implementation patterns

• Throttled re-admission as a designed, tested capability — admitting load at a measured sustainable rate and ramping while watching saturation, with the rate adjustable during the drain, since what is sustainable changes as the system recovers.
• Server-directed reconnect pacing, so the admission rate is controlled by the side that knows the capacity — and it works even with old clients in the field, which client-side backoff does not.
• Exponential backoff with full jitter in every client, where the jitter is the mechanism, not a refinement.
• Cheap resumption: session tokens and sequence numbers, so a reconnect restores rather than re-establishes.
• Deduplicate the backlog before replaying it, since a large fraction of post-incident queues are retry duplicates — a large reduction for almost no effort, and routinely skipped.
• Triage the backlog for relevance, discarding obsolete work under a pre-agreed policy.
• Prioritise by business value rather than arrival order, since FIFO processes a three-hour-old payment behind a ten-minute-old analytics event.
• Separate capacity for recovery, so live traffic is not competing with the drain — and live traffic must win.
• Cache warming before restoring full traffic, with request coalescing so simultaneous misses on one key produce a single origin fetch.
• Load-test the recovery scenario explicitly, which is a different and larger number than the steady-state test.

Industry example

The pattern appears in the largest published outages of the period. Roblox's 2021 recovery was prolonged because reconnecting a very large fleet to a cluster whose failure mode was contention under load would have reproduced the outage, requiring careful throttled re-admission. Slack's incidents involving mass client disconnection show the same shape, with the published analyses converging on server-controlled reconnection pacing and cheap session resumption as the mechanisms that matter.

The general lesson those accounts share: the mechanisms that address the amplification are more valuable than any fix for the trigger, because the amplification is what determines the duration.

Failure scenarios

• Releasing the full backlog at once, producing a self-inflicted second incident.
• Backoff without jitter, which moves the synchronised herd rather than dispersing it.
• Client-side-only backoff, leaving the server unable to influence a fleet of old clients.
• Expensive reconnection with no resumption path.
• Cold-cache stampede onto a database that has just failed over and is at its most fragile.
• Replaying into a still-broken system, producing failures that re-enqueue — unbounded work.
• Poison messages with no dead-letter limit, cycling forever and consuming recovery capacity.
• A replay throttle built during the incident, used for the first time under the worst conditions.

Trade-offs

Designing for recovery load means deliberately restoring service more slowly, keeping some users unavailable longer in order to bring everyone back sooner. That is correct and uncomfortable, and it must be a conscious product decision, because "please wait, reconnecting in 40 seconds" is worse for an individual and better for the population.

Provisioning capacity for full recovery load is the alternative and is usually uneconomic — which is precisely why admission control, cheap resumption and prioritised draining are the standard answers rather than headroom.

The trade is slower restoration and additional mechanism in exchange for a restoration that succeeds on the first attempt. For any system with persistent connections, queued work or a heavily-relied-upon cache, it is not optional — the alternative is discovering during the incident that recovery is the harder problem.

Interview question

"We fix the root cause at 03:10 and the site is still down at 06:00. Walk me through what is happening, tell me what you would do at 03:11 to make that not be the outcome, and tell me which of those things has to have been built beforehand."