Zombie Task

An orchestrator knows a task is alive because the worker says so. When those heartbeats stop — the worker was killed, the node was preempted, the network partitioned, the scheduler itself restarted — the orchestrator has to make a decision on incomplete information.

It marks the task failed and retries it. The task may still be running. A process that lost its connection to the metadata database can happily keep writing to a warehouse, a vendor API or an operational table, and now a second attempt is doing the same work beside it.

This is the concrete form of at-least-once execution in a batch platform, and it is the reason "my tasks are idempotent" is a design requirement rather than a nicety.

Why it matters

The damage is silent. Both runs report success and no alert fires, because from the platform's point of view the retry worked. What is wrong is the data: a fact table with a day's rows twice, a customer emailed twice, a payment file submitted twice.

Detection typically comes days later from a reconciliation or a complaint, by which point the duplicated partition has been aggregated into other tables. Time to detect, not frequency, is what makes this class expensive.

Implementation patterns

• Make every write a pure function of the logical date, with partition overwrite or merge on a natural key, so a second run replaces rather than appends.
• Attach idempotency keys to external calls, derived from the task identity and logical date, so the vendor de-duplicates on its side.
• Fence the output with an attempt number. Store the attempt with the result and reject a write whose attempt is lower than the recorded one, so a late zombie cannot overwrite a newer result.
• Tune the heartbeat timeout deliberately. Too short and healthy long tasks get killed; too long and recovery is slow. Timeouts of 5–10 minutes are common for tasks measured in tens of minutes.
• Prefer a scheduler that can fence at the executor level, cancelling the container rather than trusting the process to notice.

Industry example

Batch orchestrators have converged on this behaviour: Airflow has surfaced zombie detection since its earlier releases, marking tasks whose heartbeat has expired as failed and retrying them under the DAG's retry policy, and the same reconciliation exists in every scheduler that survives its own restart. The generic lesson is older than any of them: a distributed scheduler cannot distinguish a dead worker from a slow one, so it must choose between duplicate execution and stalled pipelines, and every mainstream orchestrator chooses duplicates. Kubernetes makes the same trade: when a node stops reporting, the control plane marks its pods for deletion after a grace period although those containers may still be running in production.

Failure scenarios

• Double-counted revenue after a scheduler restart, because the load appends rather than merges.
• A vendor receiving two payment instructions from one logical run.
• A retry storm: 30 interrupted tasks all retried at the same instant on a pool sized for a steady rate, causing timeouts that trigger further retries.
• Late zombie overwrite: attempt 1 finishes after attempt 2 and writes stale output over it, which is the hardest variant to reason about after the fact.
• A killed task leaving a lock or a staging table that the retry cannot clean up, so the retry fails too and the pipeline stops for a human.

Trade-offs

Aggressive heartbeat timeouts recover pipelines fast and kill healthy long-running tasks; generous ones avoid that and leave a stalled graph for longer. Idempotent design costs real effort — natural keys, merge logic, idempotency keys with the vendor — and it is the only choice that makes the trade-off disappear rather than relocating it.

When not to use it

The concept applies wherever there are heartbeats, but the engineering only pays for itself on tasks that mutate something outside the platform's control. For a graph whose every sink is a partition overwrite of a derived table, duplicate execution is a cost question, not a correctness one, and adding fencing tokens there is ceremony. Find the handful of tasks that touch an operational database, a vendor or a notification channel and spend the effort on those.

Interview question

Q: Your orchestrator's scheduler crashed at 02:40 and restarted at 03:00. Finance says one day of revenue is double. Nothing failed. Walk me through what happened and what you would change so it cannot recur.

What a strong answer covers: heartbeat expiry causing retry of tasks that were still running; at-least-once as the orchestrator's designed behaviour rather than a bug; the difference between appending and merging sinks; idempotency keys for external systems; attempt fencing for late-finishing zombies; alerting on the scheduler heartbeat because task-level alerts stay green throughout; and not proposing "make the scheduler highly available" as the primary fix, since that reduces frequency and changes nothing about the semantics.

Quick check

Quiz: Why does a zombie task produce no alert? — Both the retry and the original write succeed, so every task reports success; only the data is wrong.

Flashcard: Which tasks in a DAG actually need idempotency work? — The ones that mutate state outside the platform: operational databases, vendor APIs, notifications. Derived-table overwrites are already safe to repeat.