Airflow: A Scheduler Born From Pipeline Sprawl
also called Apache Airflow
Airbnb built Airflow because cron cannot express dependencies, and a data platform's failures are mostly dependency failures.
The problem
Data pipelines are graphs, and cron is a clock. A job scheduled for 02:00 runs at 02:00 whether or not its inputs arrived.
The failure this produces is the characteristic one of data platforms: the report is generated successfully, from yesterday's data, with today's date on it. Nothing errors. The number is simply wrong, and it is wrong in a way that looks entirely normal until someone reconciles it weeks later.
At Airbnb, with a growing number of interdependent pipelines feeding analytics and product features, managing that dependency graph through scheduling times and informal knowledge stopped scaling.
What they did
Airflow, created at Airbnb in 2014 and later donated to the Apache Foundation, models pipelines as directed acyclic graphs of tasks with explicit dependencies. A task runs when its upstream tasks have succeeded, not when a clock says so.
The properties that made it widely adopted: dependency-driven execution; per-task retry with backoff; parameterisation by logical date rather than execution time, so a backfill uses the correct business date and is the same code as a forward run; and visibility of the graph, so the impact of a failure can be assessed before deciding how urgent it is.
The trade-off
An orchestrator is a shared, stateful, high-consequence component with its own scaling and failure characteristics — and a scheduler outage stops the entire data platform. It also introduces a programming model that must be learned, and a common failure is teams writing tasks that are not idempotent, so a retry duplicates rather than replaces.
The transferable lesson
Idempotent, partition-scoped tasks are the property that makes an orchestrated pipeline recoverable. A task that appends rather than replaces will duplicate on retry — and since retries are automatic, the duplication is silent and periodic.
The design rule: each task should own a partition and replace it. Then re-running any period is safe, backfill is the same code as the forward path, and correcting a defect six months later is a parameter change rather than a project. Pipelines that lack this property do not get corrected when they are found to be wrong, which is the real cost.