A flash sale causes an incident. Two hours later the fault is fixed but 4 million jobs are backed up and notifications are hours late. Walk through recovery and prevention.

What is being tested

Whether you recognise that after the fault is fixed the backlog is the outage, and whether you can prioritise under pressure rather than simply adding workers.

Recovery, in order

1. Establish what is in the queue. Not all 4 million jobs are equal. Typically it will be a mix of order confirmations (time-critical, customer-visible), inventory sync (business-critical), analytics events (worthless if late, harmless if dropped), and retries of jobs that already succeeded.

2. Triage before scaling. Adding workers to a mixed queue processes the backlog in arrival order, which means the analytics events queued at minute 3 are processed before the order confirmations queued at minute 90. That is exactly backwards.

• Drain time-critical classes first, into their own worker pool.
• Consider discarding what has expired. A "your order is confirmed" notification three hours late may be worse than none, and a stale inventory sync is superseded by the next one. Dropping work is a legitimate and underused recovery tool, and it needs a pre-agreed policy so it is not debated at 2am.
• Deduplicate. Retries of already-successful jobs may be a large fraction of the backlog.

3. Scale carefully, watching the downstream. Ten times the workers means ten times the load on the database and on every third-party API the jobs call. The classic second outage is caused by recovery: the backlog is drained at full speed into a downstream that then falls over, requeueing everything. Rate-limit the drain to what downstream systems can absorb.

4. Communicate. Give support an expected drain time and a way to expedite an individual customer's job.

Prevention

Separate queues per class of work, with separate worker pools. This is the single highest-value change. A shared queue means a flood of low-value jobs starves urgent ones and one slow job type consumes every worker. Separate pools are a bulkhead. GitHub's background job system is organised this way for exactly this reason — nothing triggered by a push may be able to starve anything else.

Bounded queues with an explicit full policy. An unbounded queue converts a throughput problem into a memory problem and then into an outage, and it makes recovery time unbounded. When full: shed, reject at the edge, or apply backpressure to the producer — but decide which, in advance.

Load shedding at the edge during a spike, rejecting cheaply rather than accepting work you cannot do. Accepting work you cannot complete is a promise you are going to break later, expensively.

Alert on queue age, not queue depth. Depth is meaningless without throughput; the age of the oldest unprocessed message is directly meaningful and maps to customer experience.

Dead-letter queue with alerting. Poison messages must leave the main queue after N attempts, and an unmonitored DLQ is a silent data-loss mechanism.

Idempotent handlers, so the aggressive requeuing that happens during an incident is harmless.

The architectural point

Queues absorb bursts; they do not create capacity. If the sustained arrival rate exceeds the sustained processing rate, the queue only changes when you fail — and it converts a fast, visible failure into a slow, invisible one that is much harder to recover from.