Workflow Schedulers
Tools that run interdependent tasks in the right order with retries and visibility — where the value is the explicit dependency graph, not the schedule.
Definition
A workflow scheduler executes tasks according to a declared dependency graph, handling ordering, retries, parallelism, backfill and observability.
Why the dependency graph is the point
Cron expresses a schedule; it cannot express a dependency. "Run at 3am and hope the upstream finished" does not merely delay under stress — it produces silently incorrect results, because the downstream runs on incomplete data and reports success.
Representing the pipeline as an explicit graph in code makes the dependencies reviewable, testable and visible, and it is the single reason these tools exist.
The properties that matter
- Idempotent, parameterised tasks. Re-running for a given date must produce the same result and replace rather than append, or backfills double-count. This is the property that makes recovery from a logic bug a one-day job rather than a week.
- Backfill as a first-class operation, with controlled parallelism so it does not starve the daily run or overwhelm the warehouse. A logic fix always needs history reprocessed.
- Retries with backoff, and a defined terminal state that alerts.
- Data quality checks as tasks in the graph, failing the pipeline rather than propagating bad data to a dashboard someone will act on.
- Freshness SLAs with alerting, so a stalled pipeline is found by monitoring rather than by a business user.
- Lineage, so when a table is wrong you know what derives from it.
What to watch for
Scheduler as a single point of failure. Everything depends on it; it needs the same availability treatment as any critical service.
Business logic migrating into the orchestrator. Tasks should invoke logic that lives elsewhere and is independently testable. An orchestrator containing transformations becomes an untestable system nobody owns.
Tasks that are not idempotent, which makes every retry and backfill dangerous.
Graphs too large to comprehend. Split by domain with clear interfaces between them.
Failure scenarios
- Silent partial success — a task processes 60% of its input, succeeds, and nothing counts the difference.
- Dependencies expressed as timing, producing wrong results under stress.
- No backfill capability, so a logic fix is a bespoke script.
- No freshness alerting, so staleness is reported by a user.
Interview question
"Why is cron insufficient for a pipeline of interdependent data jobs?"