Why is cron insufficient for a pipeline of interdependent data jobs?
Show the full answer Hide the answer
What is being tested
Whether you recognise that timing-based sequencing produces incorrect results rather than merely late ones.
The mechanism
"Run at 3am and hope the upstream finished by then" works until the upstream is slow. Then the downstream runs on incomplete data, produces a result, and reports success.
That is the crucial point: it does not fail. It produces a wrong number that someone acts on, and the failure is discovered — if at all — weeks later when a total does not reconcile.
An explicit dependency graph makes the downstream wait for its inputs, so a slow upstream produces a delay rather than a wrong answer.
The other properties a scheduler provides
- Idempotent, parameterised tasks. Re-running for a given date produces the same result and replaces rather than appends — which is what makes a backfill safe.
- Backfill as a first-class operation, with controlled parallelism so it does not starve the daily run. A logic fix always needs history reprocessed, and without this it is a bespoke script.
- Retries with backoff and a terminal state that alerts.
- Data quality checks as tasks in the graph, failing the pipeline rather than propagating bad data to a dashboard someone will act on.
- Freshness SLAs with alerting, so a stalled pipeline is found by monitoring rather than by a business user.
- Lineage, so when a table is wrong you know what derives from it.
What to watch for
The scheduler as a single point of failure — everything depends on it, so it needs the availability treatment of any critical service.
Business logic migrating into the orchestrator. Tasks should invoke logic that lives elsewhere and is independently testable. An orchestrator containing transformations becomes an untestable system nobody owns.
Graphs too large to comprehend. Split by domain with clear interfaces.
The failure that persists longest
Silent partial success — a task processes 60% of its input, succeeds, and nothing counts the difference. Tasks should emit counts of records in, out and rejected, and the pipeline should fail when they do not reconcile.