A daily pipeline has been producing wrong numbers for three weeks due to a logic bug. What must be true of the architecture for the fix to be a one-day job?

What is being tested

Whether you understand that reprocessability is an architectural property decided long before the bug, not a recovery technique.

The four properties that make this a one-day job

1. The raw data was retained. If the pipeline transformed on the way in and discarded the source, the correct numbers no longer exist anywhere and cannot be recovered. This is the single strongest argument for ELT over ETL: raw data retained is optionality, and the option is exercised every time a transformation turns out to be wrong — which is regularly.

2. Tasks are idempotent and parameterised by date. Re-running the transformation for 2026-08-04 must produce the same result as the first run and must replace rather than append. If tasks append, a backfill double-counts and you have made the problem worse. This means writes are partition-overwrites or merges keyed on the partition, never blind inserts.

3. Backfill is a first-class operation. The orchestrator can re-run a date range with the new logic, with controlled parallelism so the backfill does not starve the daily run or overwhelm the warehouse. If backfilling means someone writing a bespoke script, this is a week, not a day.

4. Transformations are version-controlled code with a dependency graph. You can determine exactly which downstream tables and dashboards derive from the corrected table, and re-run them in order. Without lineage, you fix one table and discover three weeks later that a dashboard fed from a copy is still wrong.

This is the practical value of representing a pipeline as an explicit dependency graph in code — the insight behind Airbnb's development of a general workflow orchestrator. Cron could express the schedule but not the dependencies, could not retry sensibly, and made nothing visible. Dependencies expressed as timing ("run at 3am and hope the upstream finished") do not merely delay under stress; they produce silently incorrect results.

The plan for the day

1. Fix the transformation, with a test that reproduces the bug.
2. Determine the affected date range and the full downstream dependency set from lineage.
3. Backfill into a staging location, not over the live tables.
4. Compare old and new for a sample of dates — expect and explain the differences before switching.
5. Swap in the corrected data, then re-run downstream in dependency order.
6. Communicate, especially to anyone who made a decision on the wrong numbers.

What would have caught it in three days rather than three weeks

Data quality tests as tasks in the graph, which fail the pipeline. Row-count bounds, null rates, referential integrity, and — most valuable — distribution and volume checks against recent history. A number that moves 30% overnight is either a real event or a bug, and both deserve an alert.

Also worth having: reconciliation against an independent source, so warehouse revenue is compared with the operational system's total daily. That catches whole categories of error that no unit test would.

The uncomfortable point

Three weeks means nobody was checking. The architectural fixes above make recovery cheap; only monitoring makes detection fast, and detection is where the actual damage was done — every decision made on those numbers during three weeks.