Backfills and Reprocessing

A bug is found in a transformation that has run for three months. Fixing the code is an hour. Recomputing three months of derived data without corrupting the live pipeline, overwhelming shared infrastructure, or producing results that disagree with what downstream systems already consumed is the actual work, and pipelines are rarely designed for it.

Why it is not just a fast run

Volume arrives at once. A pipeline sized for a day's data processing ninety days of it will exhaust memory, saturate the network, or hit rate limits on every dependency it calls. Backfills need explicit throttling and chunking, usually by partition, with bounded parallelism.

The world has changed. A backfill of last quarter runs against today's schemas, today's reference data, today's third-party services. If the transformation joins against a dimension table that has since been updated, the recomputed output reflects current dimension values rather than historical ones, so the backfill produces something that is neither the old result nor the correct one. Reproducing history requires historical inputs, which requires those inputs to have been versioned, which is a decision made long before the backfill.

Downstream consumers already read the old values. Reports were published, models were trained, decisions were made. Silently replacing the data changes what those artefacts would produce without changing the artefacts, so the plan must include what happens to consumers rather than only what happens to the table.

Making a pipeline backfillable

Idempotent, partition-scoped writes. Each run must fully replace the output partition for its input range, so rerunning any partition converges to the same state. Append-only outputs are not backfillable without a separate deletion step, which is where duplicates come from.

Explicit input ranges. The pipeline takes the interval to process as a parameter, rather than reading "now" or "the last watermark". A pipeline that computes its own window cannot be asked to compute a different one.

No hidden dependence on current state. Every input must be addressable as of the processing interval, including reference data and configuration. This is why versioned dimension tables and point-in-time lookups matter well beyond feature stores.

Isolated compute. Backfills should run in a separate queue or cluster from production, so a large recompute cannot starve the live pipeline. Sharing infrastructure means every backfill is also an availability risk.

When it breaks

Cascading recomputation is usually forgotten. Fixing a table means every derived table needs rebuilding, in dependency order. Without lineage that ordering is reconstructed by hand and something is missed, so the stack ends up in a state where two tables disagree and the disagreement is not obviously attributable.

Streaming backfill is a genuinely different mode. Replaying a log from an old offset floods the pipeline at maximum throughput, which produces watermark behaviour nothing like steady state: windows advance in seconds, timers fire in bursts, and state grows far faster. Pipelines that are correct in steady state routinely fail on replay for this reason alone, and testing replay is not the same as testing the pipeline.

Late-arriving corrections must be distinguishable from the original. If a backfill rewrites values a consumer already has, the consumer needs to know which rows changed. A processing timestamp or version column on the output row is what makes incremental downstream refresh possible, and adding it after the fact means a full downstream rebuild.

Cost is easy to underestimate. A three-month backfill of a pipeline costing a few dollars a day is ninety days of compute compressed into hours, frequently at on-demand rather than reserved rates. Estimating it before running, and running one partition first to measure, avoids a bill that arrives as a surprise.