A data platform's batch jobs have complex interdependencies and a failure early in the night cascades. What orchestration properties matter?

The properties

1. Dependencies declared, not implied by schedule. Jobs scheduled by time and assumed to complete in order is the most common design and the most fragile — a slow upstream job means a downstream job runs on stale or partial data, silently. Dependencies must be on data availability, not on the clock.

2. Idempotent, re-runnable tasks. Any task must be safely re-runnable for a given logical date, producing the same result. Without this, recovery from a partial failure requires manual reasoning about what already ran.

3. Partitioned by logical date, so a rerun for one day does not reprocess everything and a backfill is a range of independent executions.

4. Explicit data-quality gates between stages, so a bad upstream result stops propagation rather than producing plausible wrong numbers downstream.

5. Retry with backoff for transient failures, and a clear distinction from permanent ones — so a transient network error does not require human intervention and a schema mismatch does not retry forever.

6. Observable lineage, so the answer to "why is this dashboard wrong" is a traversal rather than an investigation.

Handling the cascade

Isolate blast radius by domain. A failure in one source's pipeline should hold that source's downstream consumers, not the whole night. This is the same per-source isolation principle that governs multi-partner ingestion.

Prioritise recovery. Not all outputs are equal — the operational dashboard the business opens at 8 a.m. matters more than a monthly aggregate. The orchestrator should support priority so recovery serves the critical path first.

Partial completion must be visible, so consumers know a dataset is incomplete rather than assuming it is current. A stale dataset presented as current is worse than a missing one.

The backfill requirement

Backfills must not starve the regular schedule. A reprocessing job replaying months of history will consume the same capacity as the nightly run unless resources are isolated — which is the same workload-isolation principle that applies to compaction and to any large asynchronous job.