A logic bug means six months of a derived table are wrong. You must reprocess without disrupting live consumers. How?
Show the full answer Hide the answer
Establish the blast radius first
Which downstream tables, reports and extracts consumed the wrong data, and did any of it leave the organisation — a regulatory submission, a customer-facing figure, a partner file. That determines whether this is a correction or a notifiable event, and it changes the sequencing.
Then check you can reprocess at all
The feasible window is bounded by the shortest retention on the path, not by the input's. An intermediate table or topic left at a seven-day default blocks a six-month replay even when the source has a year.
If a stateful stage's state has expired, that stage must be recomputed too, which may not be possible.
The three arrangements, and how to choose
Parallel output and swap. Run the corrected logic into a separate table, verify it, then repoint consumers. Cleanest, no disruption, costs a duplicate copy. This is the default answer.
Same output with idempotent keys. Re-emit over the existing rows using a merge on a natural key. Requires that every write is idempotent — if any downstream aggregation increments rather than replaces, this double counts.
Stop, reprocess, restart. Simple and usually unacceptable on anything with live consumers.
Verify before swapping
Reconcile the corrected output against an independent source for a sample of periods, and compare row counts and key aggregates against the old version. Expect differences and be able to explain each one — an unexplained difference means the correction has its own bug, which is a common and embarrassing outcome.
Communicate the restatement
Downstream consumers need to know that historical figures have changed, from when, and by how much. A silent restatement destroys trust in the platform more thoroughly than the original error did.
Afterwards
Set retention on every stage against the longest replay you might plausibly need, and write down why. That single change is what turns the next reprocessing from a project into an operation.