A clip-moderation queue for a live-streaming platform of Twitch's kind runs with a 30-second visibility timeout. During a surge, per-clip processing time rises from 8 to 45 seconds. By 14:20 the queue is draining at a third of its usual rate, most clips are reviewed two or three times, and a few are reviewed indefinitely. What failed, and which design decision allowed it?

The trigger

Processing time crossed the visibility timeout. A visibility timeout is a lease, and the broker's contract is that if the lease expires without a delete, the message is redelivered - it has no way to know the first worker is still running. At 45 seconds of work against a 30-second lease, every message is handed to a second worker 15 seconds before the first one finishes.

Why it propagated

The duplicate consumes the capacity that would have cleared the backlog, which makes processing slower still, which widens the gap between work time and lease, which produces more duplicates. It is a feedback loop whose input is latency and whose output is more latency, and it does not stabilise on its own. Effective throughput fell to about a third because roughly two of every three completions were repeat work.

The few clips reviewed indefinitely are the second failure: with no limit on redeliveries, an item that always exceeds the lease - the longest clips - loops forever and is never completed or set aside.

Why detection lagged

Every signal looked explicable. Queue depth was high, which a surge explains. Worker error rate was zero, because the workers were succeeding - twice. CPU was saturated, which a surge also explains. The two signals that name the problem are the ones nobody graphs: receive count per message, and the age of the oldest in-flight message. A receive count above 1 on a healthy queue means the lease is mis-sized, and it is visible from the first minute.

The structural fix

1. Renew the lease while working. A heartbeat that extends visibility every few seconds turns a guess about duration into a statement about liveness. A worker that dies stops extending and the message returns in seconds rather than minutes.
2. Make the unit of work smaller than the lease. Splitting a clip review into fetch, scan and decide, each under 10 seconds, removes the race instead of managing it.
3. Bound redeliveries. A redrive policy that moves a message to a dead-letter queue after a small number of receives converts an infinite loop into a triage queue with an owner.
4. Make the handler idempotent - keyed on clip id - so a duplicate is wasted work rather than a double moderation decision. This is the part that protects correctness; the rest protects throughput.

The tempting local fix, and why it is worse

Raise the timeout to five minutes. It stops the duplicates and it also means a worker that crashes holds its message invisible for five minutes, so a rolling deploy during a surge now adds a five-minute stall per in-flight item. A static lease trades duplicate work against recovery time, and under variable processing times there is no setting that is right for both. That is the argument for renewal rather than for a bigger number.

The general lesson

A lease is a bet on how long work takes. Any bet that can be lost needs a renewal mechanism and a loss limit - here, heartbeat extension and a dead-letter threshold. The same shape appears in distributed locks, leader leases and HTTP request deadlines.

When not to add renewal

For work that is reliably sub-second, a static timeout a few multiples above the expected duration plus idempotent handlers is correct, and heartbeat renewal is machinery nobody needs. The lease only needs renewal when the work's duration distribution has a long tail - which is exactly what media processing has and what a small CRUD job does not.