Pipeline Orchestration
Coordinating the execution of data tasks by dependency rather than by clock, with retries, backfill and observability built in.
Cron schedules a job at a time. An orchestrator runs a job when its inputs are ready, which is a materially different thing once pipelines have dependencies — and it is why "the report was wrong because the upstream load had not finished" stops happening.
What an orchestrator supplies that a scheduler does not: a dependency graph, per-task retries with backoff, backfill across historical periods, parameterisation by execution date so a task is a pure function of its window, observability of run history, and alerting on failure and on lateness.
The design property that matters most is idempotency per partition. A task should be safe to re-run for a given date and produce the same result — which means writing to a partition rather than appending, and deriving all inputs from the execution date rather than from "now". Without it, backfill is dangerous and retries are worse than failures.
Airflow, Dagster and Prefect are the common tools; dbt handles transformation within the warehouse and is usually orchestrated by one of them. The choice matters less than the discipline: data dependencies, not clock times, and every task idempotent for its window.