advanced 2 min answer

A logic bug means six months of a derived table are wrong. You must reprocess without disrupting live consumers. How?

data-platformbackfillreprocessingcorrectness
Show the full answer Hide the answer

Establish the blast radius first

Which downstream tables, reports and extracts consumed the wrong data, and did any of it leave the organisation — a regulatory submission, a customer-facing figure, a partner file. That determines whether this is a correction or a notifiable event, and it changes the sequencing.

Then check you can reprocess at all

The feasible window is bounded by the shortest retention on the path, not by the input's. An intermediate table or topic left at a seven-day default blocks a six-month replay even when the source has a year.

If a stateful stage's state has expired, that stage must be recomputed too, which may not be possible.

The three arrangements, and how to choose

Parallel output and swap. Run the corrected logic into a separate table, verify it, then repoint consumers. Cleanest, no disruption, costs a duplicate copy. This is the default answer.

Same output with idempotent keys. Re-emit over the existing rows using a merge on a natural key. Requires that every write is idempotent — if any downstream aggregation increments rather than replaces, this double counts.

Stop, reprocess, restart. Simple and usually unacceptable on anything with live consumers.

Verify before swapping

Reconcile the corrected output against an independent source for a sample of periods, and compare row counts and key aggregates against the old version. Expect differences and be able to explain each one — an unexplained difference means the correction has its own bug, which is a common and embarrassing outcome.

Communicate the restatement

Downstream consumers need to know that historical figures have changed, from when, and by how much. A silent restatement destroys trust in the platform more thoroughly than the original error did.

Afterwards

Set retention on every stage against the longest replay you might plausibly need, and write down why. That single change is what turns the next reprocessing from a project into an operation.