A bug in a streaming job produced wrong results for three weeks. How should the backfill be designed so it does not cause a second incident?

The three questions that determine the approach

1. Is the output store idempotent for this data? If reprocessing overwrites by key, the backfill is simply a re-run. If it appends or increments, reprocessing double-counts and the old output must be removed or isolated first.

2. Is the input still available? Reprocessing requires the source log to retain three weeks of history — which is exactly what tiered storage makes affordable and what a short retention makes impossible. If the input is gone, the backfill is a reconstruction from another source, which is a different and harder project.

3. Who is consuming the wrong output right now? They must either be told, or served from a corrected view, and this is a communication decision that belongs to the business.

The design

Write to a new output version rather than correcting in place. Run the fixed job over the affected period into a parallel table or partition set, validate it, then swap the serving pointer atomically — so readers see the old or the new version and never a mixture, and rollback is a pointer flip rather than another three-week job.

Validate before promotion: row counts against expectations, distribution comparison with the pre-bug period, and a canary set of records whose corrected values are known independently. The failure being guarded against is a backfill that completes successfully and produces different wrong data, which no infrastructure alarm detects.

Not causing a second incident

• Throttle the backfill. Three weeks of events replayed at full speed is a load spike of enormous magnitude — the same shape as a post-incident backlog replay, and it will saturate the output store and every downstream consumer.
• Use separate compute and, where possible, separate downstream capacity, so live processing is not competing with the backfill. Live must win.
• Watch downstream saturation and make the rate adjustable during the run, since the sustainable rate changes as the system absorbs load.
• Do not emit backfill output onto the live event stream unless downstream consumers are explicitly designed for it — otherwise every downstream system receives three weeks of events in an hour, and the incident propagates outward.
• Isolate side effects. A reprocessed order event that re-triggers notifications, emails or payments is the classic and severe backfill failure. Every side-effect boundary must recognise reprocessed events and refuse.

Making it routine rather than heroic

• Design jobs to be idempotent and replayable from the start, keyed by event identifier and event time.
• Version the outputs, so a corrected run is a new version rather than an in-place mutation.
• Retain input long enough to reprocess, which is a deliberate retention decision with a cost.
• Separate event time from processing time throughout, so a backfill produces correct windowing rather than assigning three weeks of history to today.
• Keep a tested throttled replay mechanism, because one built during the incident is being used for the first time under the worst conditions.

A pipeline that cannot be reprocessed safely has a defect independent of any bug in its logic — and it will be discovered at the moment a bug appears, which is the worst time to learn it.