Backfill
Re-running a pipeline over historical periods to populate new data or correct a past error, and the operation that proves whether a pipeline is well designed.
Needed whenever a new column is added, a bug is found in transformation logic, a source is corrected, or a new dataset must cover history.
It is a good stress test of the design because it requires properties that are easy to omit. Tasks
must be idempotent per partition, so re-running a date overwrites cleanly rather than duplicating.
Inputs must derive from the execution date, not from now() — a transformation containing
CURRENT_DATE cannot be backfilled correctly and the error is silent. Source data must still be
available for the period, which is a retention question that should have been asked at design time.
Operational care matters too. A backfill of two years at full parallelism will saturate the warehouse and disrupt production workloads, so it needs throttling and preferably a separate compute pool. Backfills are long, so they must be resumable rather than restarting from the beginning.
The strongest test to apply to any pipeline: can it be re-run for an arbitrary past date, safely, right now? If the answer is no, that is a defect, and it will be discovered at the worst moment — when a bug has already corrupted history.