A bug is found in a stream processor that has been running for a month. How should the correction be handled?

What must be true for reprocessing to be possible at all

• The source log retained for long enough. If retention is seven days and the bug is a month old, the data to reprocess from does not exist — and this is discovered at the worst moment. Retention is therefore a correctness decision rather than a cost decision.
• Downstream idempotency, so reprocessing does not duplicate effects.
• A deterministic processor, or reprocessing produces different results from the original — which for a processor calling external services or using wall-clock time it will.

The two approaches

Reprocess in place: reset the consumer's offset and replay. Simple, and it produces a period where the downstream state is inconsistent while the corrected data catches up — and consumers see values change without explanation.

Reprocess to a parallel output, then switch consumers when it is complete and verified. More infrastructure, no inconsistent window, and it allows verification before anything is exposed. This is the safer choice for anything a decision is made on, and it is the kappa architecture's practical argument.

What must be handled either way

• Rate limiting the replay, since a month of data at full speed will overwhelm downstream systems that are sized for the live rate.
• Distinguishing replayed from live events downstream, so anything with an external effect — a notification, a payment, a webhook — is suppressed. Replaying a month of notifications is the classic self-inflicted incident here.
• Deciding whether corrections are visible. If a value is restated, consumers must tolerate it — and most are built assuming they will not have to.

The design decision that makes this tractable

Separate the external effects from the computation. A processor that computes state and a separate component that acts on state changes can be reprocessed safely; one that sends emails inline cannot be replayed at all without a suppression mechanism.

That separation costs little to build and is the difference between a correctable pipeline and one where a bug's damage is permanent.