Spotify built Luigi (open-sourced in 2012) to run chains of Hadoop batch jobs, and decided that a task is complete when its output target exists, rather than when a scheduler's database says it ran. What problem did that choice solve, what did it cost, and when would you not copy it?

The situation they were in

Spotify's early data jobs were long chains: log aggregation feeding royalty reports, charts and recommendation inputs, each step reading files written by the one before. Chains broke halfway all the time (a node died, a cluster hiccuped), and the operational question after every failure was the same: what has actually been done, and what must be re-run?

What Luigi chose

In Luigi a task declares an output(), such as a file path or a table partition, and a requires() listing upstream tasks. Before running, Luigi asks each task's output whether it exists. If it does, the task is skipped and its dependencies are never walked. Re-running the whole pipeline after a failure is therefore the recovery procedure: completeness is read from the data, not from a metadata store.

Why it fit

• No scheduler database to drift out of sync. An orchestrator that tracks run state separately from the data can believe a task succeeded when the output was deleted, or failed when the output landed before the worker died.
• Re-running is safe by construction, provided tasks write atomically (a temporary path, then rename). A half-written output is never mistaken for a finished one.
• Backfills are trivial: ask for the targets over a date range and only the missing ones run.

What it cost

• Luigi shipped without a built-in cron-style trigger; teams ran it from cron or another scheduler, so time-driven scheduling lived elsewhere.
• Existence is a weak completeness test. An empty file or a partition written from a partial upstream exists and is wrong. The pattern needs success markers or row-count checks to be safe.
• A task that depends on external state (an API call, a mutable table) has no natural target, so the model fits poorly.
• Scale limits: a central scheduler resolving large dependency graphs by repeated existence checks against remote storage becomes slow, one reason later systems keep state in a database.

What a modern equivalent looks like

Table formats now make "output exists" a transactionally committed snapshot, and orchestrators such as Airflow and Dagster track run state and asset materialisation together. The Luigi lesson survives as a rule: make completion a property of the data (an atomic commit or a manifest) so a retry can be decided without trusting a log.

Where copying it would be a mistake

If your tasks write to mutable tables, call external services or have side effects other than files, target-existence is not a valid completion signal and you will skip work that was never done. Prefer explicit run state plus idempotent writes. And with a handful of jobs, a scheduler's own retry and status UI is worth more than purity.

Common weak answers

• "Luigi is just an older Airflow." The difference is where truth lives, not the age.
• "Skipping existing outputs is risky." It is only risky when writes are non-atomic or existence is the only check.