A transformation bug means months of derived data are wrong. What makes reprocessing feasible, and what breaks it?

What makes it feasible

1. Raw data retained. The original events, unmodified, for long enough to replay. Without this the error is permanent, and no amount of pipeline sophistication recovers it. This is the property that justifies a raw retention layer, whatever it is called.

2. Deterministic transformation. Replaying the same input through the same logic must produce the same output. Dependence on wall-clock time, on the current state of a mutable lookup, or on processing order breaks this.

3. Partitioned, replaceable output. Reprocessing a period writes to that period's partition, replacing it — so a backfill is a range of independent executions rather than a global rebuild.

4. Versioned logic, so the output records which version produced it and a partial reprocess is identifiable.

5. Resource isolation. A replay of months of history will starve the live path if they share capacity — which is the most common contention incident on a data platform, and it means backfills need their own compute or a strict priority policy.

What breaks it

• Retention shorter than the discovery delay. A bug found after three months, with two months of retention, cannot be fully corrected. Retention must exceed a realistic time-to-discovery, not just an operational window.
• Non-deterministic transformations, particularly enrichment against a mutable lookup whose current state differs from its state at the time — which produces a reprocessed result that is different rather than corrected.
• Append-only outputs, where reprocessing duplicates rather than replaces.
• Downstream consumers that already acted on the wrong data — sent emails, made payments, trained models. Reprocessing corrects the data and not the actions.
• Backfill competing with live processing, turning a correction into an outage.

The consumer question that must be answered

What happens downstream when a period is reprocessed? Consumers that cached, aggregated further, or trained on the wrong data need to know. Without a change notification or a version marker, the correction propagates partially — and partial correction is frequently worse than none, because the inconsistency is now undetectable.