A streaming-video platform in the mould of Hotstar runs about 900 hourly batch tasks on a workflow scheduler with catch-up enabled. The scheduler is down for 40 minutes at 02:10, then returns. What happens in the first ten minutes after it returns, and what stops it from hurting production?

Second by second

The scheduler comes back and reads its state: for each pipeline, the schedule says runs were due at 02:00 (late) and 03:00 is 50 minutes away, so most pipelines have one missed interval each. With catch-up on, it creates a run for every missed interval at once. Nine hundred hourly tasks each missed one run, so roughly 900 task instances become eligible in the same second.

Workers start pulling. Whatever limits exist now decide the outcome. With no pool or concurrency limits, 900 jobs hit the warehouse and the source databases simultaneously: connection limits trip on the sources, the warehouse queues, and tasks begin to time out.

Where it amplifies

Retries. Timed-out tasks retry on their own schedule, adding load to a system already past capacity. Because the 03:00 interval's tasks also arrive on time, the system now carries missed work plus fresh work on top. If tasks are not idempotent, a retried append duplicates rows.

Tasks in the same dependency chain also wake together, so downstream tasks sit in a queue behind upstream ones occupying all slots, a classic head-of-line block.

What the user sees

Dashboards that normally refresh at :05 are 30 to 90 minutes late for the next two hours, and a few show duplicated totals if appends were retried.

What stops it

A mechanism, not monitoring:

1. Concurrency pools per shared resource (source DB, warehouse) with a small fixed slot count, so excess tasks queue instead of running.
2. A cap on active runs per pipeline so a pipeline never runs two intervals at once.
3. Idempotent tasks keyed on the logical interval, so a retry overwrites its own partition instead of appending.
4. Priority weights so business-critical pipelines drain first.
5. Catch-up disabled for pipelines where only the latest interval matters, such as a "current snapshot" refresh.

What would have to be true for self-healing

Pools sized to what the shared systems can absorb, idempotent writes, and backoff on retries. Then the 40 minutes of outage becomes about 40 minutes of extra queue, drained in roughly the time it takes to run the missed work serially through the pool.

When catch-up is the wrong default

For hourly metric refreshes that overwrite a snapshot, running 12 missed intervals is pure waste; run once with the latest data. Catch-up earns its keep for append-only partitions where every interval is needed.

Common weak answers

• "Add alerts for a scheduler outage." The alert arrives, and the storm still happens.
• "Turn off retries." That trades a storm for permanent gaps.