The batch orchestrator's scheduler process dies at 02:40 with about 30 tasks already running on workers. It is restarted at 03:00. The nightly load includes a MERGE into a revenue table and a push to a payments vendor. What happens between 02:40 and 03:15?

Minute by minute

02:40 — the workers do not care. Tasks already dispatched keep running; they are separate processes holding their own database connections. Nothing they are doing stops. What stops is scheduling: no new task is queued, and no completion is recorded, because the component that writes state transitions is gone.

02:40 to 03:00 — the silent window. Tasks finish and their success is either written directly to the metadata database by the worker, or lost, depending on the orchestrator's design. Downstream tasks do not start. The graph appears frozen at 02:40, which is the first misleading signal: an operator reading the UI sees running tasks, not a dead scheduler.

03:00 — the restart, and the dangerous part. The scheduler reconciles: for every task marked running, is there still a live worker heartbeat? Tasks whose heartbeat has expired are marked failed and retried under the normal retry policy. This is where at-least-once execution stops being a theoretical property.

03:00 to 03:15 — the second run. The MERGE re-runs, and a 40-minute load now costs 80 minutes of warehouse compute. If it is keyed on a natural key and writes deterministically, the second run is a no-op and nobody notices. If it inserts with a generated surrogate key or appends to a fact table, revenue is now double-counted for one partition, every downstream aggregate is wrong, and every job succeeded. The vendor push re-runs too, and unless the request carries an idempotency key the vendor may see two payment instructions.

Where it amplifies

The reconciliation retry hits every interrupted task at once, so 30 tasks start simultaneously on a worker pool sized for a steady rate. The load spike causes timeouts, the timeouts cause more retries, and the run finishes late enough to miss the 06:00 reporting SLA even though no single task failed twice.

What stops it

• Idempotent writes by construction: every task is a pure function of its logical date, and the write is a partition overwrite or a merge on a natural key. Then a duplicate run is uninteresting, which is the only durable answer.
• Idempotency keys on every external call, derived from the task's logical date and target.
• Run-level fencing: a monotonically increasing attempt number stored with the output, so a late-finishing zombie from attempt 1 cannot overwrite attempt 2's result.
• Alert on the scheduler heartbeat itself, not on task failures. A dead scheduler produces no failures — that is precisely why it goes unnoticed for twenty minutes.

When this is the wrong thing to worry about

For a pipeline whose sinks are all partition overwrites of derived tables, this whole class of problem is already solved and adding fencing logic is ceremony. The risk lives entirely in tasks that mutate something outside the orchestrator's control: an operational database, a vendor API, an email send. Find those and make them idempotent; the rest can be replayed freely.