Orchestration, Dependencies and Idempotency

An orchestrator is often described as a scheduler, which understates it. Cron schedules. An orchestrator manages a directed acyclic graph of tasks with dependencies, retries failures, tracks which logical intervals have succeeded, and provides the vocabulary in which a data platform's reliability is expressed. The design decisions that matter are about dependencies and idempotency, not about the cron expression.

Time-based versus data-aware scheduling

The traditional model runs a DAG on a schedule and assumes upstream data is ready. It is simple and it encodes a guess: "the source system usually finishes by 02:00, so we start at 03:00". When the source is late, the job runs on incomplete data and produces a wrong answer that looks like a right answer, which is the worst available failure mode.

Sensors partially address this by polling for a condition before proceeding, at the cost of holding a worker slot while waiting. The better model is data-aware scheduling, where a task declares the datasets it consumes and produces and the orchestrator triggers it when its inputs are actually updated. This makes the dependency graph a property of data rather than of clocks, and it makes lineage a byproduct of the scheduling metadata rather than a separate system.

Why idempotency is not optional

Orchestrators retry. Tasks fail on transient infrastructure problems, are rerun manually after a fix, and are backfilled across historical intervals. Every task will run more than once for the same logical interval, and if a rerun produces different or duplicated output, the platform has no recovery mechanism at all.

Concretely: a task should fully replace the output for its interval, not append to it. Given the same interval and the same inputs it should produce the same result. It should not depend on the current time except through the interval it was given, and it should not accumulate into shared mutable state.

This is why the execution interval, rather than the wall-clock start time, is the parameter tasks are written against. A task that computes today() internally cannot be rerun for last Tuesday, which means it cannot be backfilled and cannot be safely retried after a delay.

When it breaks

Sensors deadlock the pool. Many waiting sensors occupy worker slots, so the tasks that would satisfy their conditions cannot start. Deferrable or asynchronous sensors that release their slot while waiting exist for this reason, and pipelines using blocking sensors at scale eventually hit it.

Retry policy interacts with partial failure. A task that writes half its output and fails will, on retry, write the rest and leave duplicates unless the write is a full partition replacement. Retries are only safe for tasks whose failure leaves no partial effect, and marking a non-idempotent task as retryable is a common and damaging misconfiguration.

The DAG becomes the architecture. Deep dependency chains mean a failure early on blocks everything behind it, and a single 500-task DAG is impossible to reason about or to partially rerun. Splitting into smaller DAGs connected by dataset dependencies keeps blast radius bounded, at the cost of losing a single global view.

SLAs need to be on data, not on tasks. "The task completed" is not the same as "the data is correct and complete". Freshness and quality checks belong in the graph as tasks that can fail, so that a downstream consumer waiting on the dataset is blocked by a quality failure rather than proceeding on bad data that a green task run implied was fine.

Orchestrator state is a single point of failure. The metadata database holding run history and task state is the platform's memory. Losing it means losing the record of which intervals succeeded, and rebuilding that by inspecting outputs is possible only if the outputs are partition-addressable, which is another reason the idempotency discipline pays for itself.