A background-job system has millions of queued jobs after an incident, and replaying them all at once will crash the databases. How should recovery be sequenced?

Why the naive replay fails

The backlog accumulated over hours; releasing it takes minutes. The instantaneous rate on release is far above normal throughput, because it is the accumulated arrival rate compressed into however fast the workers can go.

Downstream systems are sized for normal rate. The recovery attempt becomes a second, self-inflicted incident — and this one occurs while the team is already tired and the system is already fragile.

The sequence

1. Stop the bleeding first. Ensure the original cause is resolved and the queue is no longer growing. A replay into a still-broken system produces failures that re-enqueue, which is unbounded work.

2. Deduplicate before replaying. A large fraction of a post-incident backlog is frequently duplicates — the same job enqueued by retries during the failure. Deduplicating first can reduce the work by a large factor for almost no effort, and it is routinely skipped.

3. Triage what is still relevant. Many queued jobs are obsolete: a notification about a state that has since changed, a cache refresh superseded by a later one, a job for an entity that has been deleted. Discarding these is legitimate and requires a business decision about which categories are safe to drop — a decision that should be made deliberately rather than by processing everything.

4. Prioritise by business value, not by arrival order. FIFO is wrong here. A payment from three hours ago matters more than an analytics event from ten minutes ago, and the queue's natural ordering will process them in the wrong sequence. Establish priority classes and drain the critical ones first.

5. Throttle the replay to a rate the downstream can absorb, measured rather than guessed — and ramp it gradually while watching downstream saturation signals. The replay rate is a control input and must be adjustable during the drain, because the sustainable rate changes as the system recovers.

6. Use separate capacity for the replay where possible — a dedicated worker pool and, ideally, separate downstream capacity — so that live traffic is not competing with recovery. Live traffic must win.

7. Handle the poison messages. An incident frequently produces jobs that will never succeed. Without a dead-letter path with a retry limit, they cycle forever and consume the capacity needed for the rest.

What should have existed beforehand

• A bounded queue with a defined overflow behaviour, so the backlog has a maximum size chosen in advance.
• Priority classes, established before the incident rather than during it.
• Job idempotency, so replay is safe and deduplication is possible.
• An expiry on jobs, so obsolete work discards itself.
• A throttle-able replay mechanism, tested — a replay control built during the incident is being used for the first time under the worst conditions.
• Load-tested downstream capacity for the recovery scenario, which is a different and larger number than the steady-state one.

Recovery load is qualitatively different from steady-state load and must be capacity-planned separately — which is the general lesson, and applies equally to reconnect storms, cache warming and failover.