practice

Backfill

Reprocessing historical periods through a pipeline after fixing a defect or adding a field, at a scale the pipeline was not sized for.

pipelinesreprocessingoperations

Backfill is where pipeline design is tested, because it takes a system built for one day's volume and asks it to process three years in an afternoon.

The properties that make it survivable are decided long before it is needed. Idempotent, partition scoped writes — each run replaces the partition it owns — so re-running any period is safe and repeatable. Parameterisation by logical date rather than execution time, so historical runs use the correct business date rather than today's. Retained raw input, since a backfill is only possible if the source data still exists in the state it arrived. And concurrency limits, because a thousand parallel historical runs will exhaust the warehouse, saturate the source, or blow the month's budget in an hour.

The consequences downstream are the part that gets missed. A backfill rewrites history, so downstream aggregates must be recomputed too, dashboards will change numbers people have already acted on, and any consumer that has exported a snapshot now disagrees with the system of record. Communicating a backfill is as much of the work as running it.