Logical Date Partitioning

A batch task is parameterised by the logical period it processes — a day, an hour — rather than by when it happens to run. Its output is written to the partition for that period, replacing whatever was there.

That single property makes reruns safe: re-executing a task for a given date produces the same result, whether it is the first run, a retry after a failure, or a backfill two years later.

Why it is the foundational property

Without it, recovery from a partial failure requires a human to reason about what already ran and what state it left. With it, recovery is re-running the affected dates, and a backfill is a range of independent executions that can proceed in parallel.

It also makes the dependency model tractable: a downstream task for a date depends on its upstream tasks for that date, which is a clear condition rather than an assumption about timing.

Implementation patterns

• Output written to the date's partition, replacing it, rather than appended. Append-based tasks are not idempotent — a rerun duplicates.
• Inputs selected by the logical date, not by "everything since last time", which makes the task depend on hidden state.
• Dependencies on data availability for that date, not on a wall-clock schedule. Time-based dependencies are the source of most silent staleness, because a downstream job runs on whatever is present.
• Late-arriving data handled by a defined policy — a rerun window, or a separate correction path — rather than silently missed.
• Backfills resourced separately, since a replay of months of history will otherwise consume the regular schedule's capacity.
• Partial completion visible, so consumers know a dataset is incomplete rather than assuming it is current — a stale dataset presented as current is worse than a missing one.

Industry example

Data platforms whose nightly failures cascade almost always lack this. Jobs are scheduled by time, assume upstream completion, and append rather than replace. A slow upstream job therefore produces downstream results computed from partial data, with no error anywhere — and the discrepancy is found days later in a report.

Adding logical-date partitioning converts that class of failure into a visible dependency that either holds or does not, and converts recovery from an investigation into a rerun.

The same reasoning applies to any pipeline that must be reprocessed: raw data retained plus deterministic date-partitioned transformation is what makes "the transformation was wrong for three months, reprocess it" a routine operation rather than a project.

Failure scenarios

• Appending rather than replacing, so reruns duplicate.
• Tasks reading "since last run", which hides state and breaks under reruns and parallelism.
• Time-based dependencies, producing silent partial-data computation.
• No late-data policy, so records arriving after a partition is processed are lost with no signal.
• Backfills competing with the schedule, turning a reprocessing request into an incident.

Trade-offs

Date partitioning imposes a grain, and work that does not decompose by time — a full graph computation, a global ranking — fits awkwardly and needs a different pattern.

Replacing partitions also costs more write volume than appending, and for very fine grains the partition count becomes a small-file problem. The usual resolution is a coarse partition grain with finer processing inside it.

Interview question

"A job failed halfway through last night's run and you need to recover. Walk me through what happens — and tell me what property of the pipeline determines whether that is a five-minute rerun or a morning of investigation."