Replay Effect Suppression

When a bug is found in a stream processor that has been running for weeks, the correction requires reprocessing the affected period. A processor that performs external effects inline cannot be replayed at all — the replay re-sends every notification, re-calls every webhook and re-triggers every payment.

The design that makes reprocessing possible: a processor that computes state, and a separate component that acts on state changes.

Why it matters

It is the difference between a correctable pipeline and one where a bug's damage is permanent. And it costs almost nothing to build in advance while being very expensive to retrofit — by which time the bug has already been found and the reprocessing is blocked.

Implementation patterns

• Computation emits state or events; a separate component performs effects. The separation must be a process boundary, not merely a function boundary, so the effect component can be disabled during a replay.
• Distinguish replayed from live events explicitly, with a flag the effect component honours. Replaying a month of notifications is the classic self-inflicted incident in this domain.
• Rate-limit the replay, since a month of data at full speed overwhelms downstream systems sized for the live rate.
• Retain the source log long enough to reprocess from. If retention is seven days and the bug is a month old, the data does not exist — which makes retention a correctness decision rather than a cost one, and it is discovered at the worst moment.
• Prefer reprocessing to a parallel output and switching consumers when verified, over resetting offsets in place — which avoids an inconsistent window where consumers see values change without explanation.
• Ensure the processor is deterministic, since one calling external services or using wall-clock time produces different results on replay.
• Require downstream idempotency, so a replay does not duplicate effects even where suppression is imperfect.

Industry example

Event-routing and streaming platforms such as Segment and Confluent-based pipelines encounter this whenever a transformation bug is discovered late. The organisations that recover cleanly are the ones whose effects were already separated; the ones that cannot either accept the bad data permanently or accept a mass re-send, and both are bad outcomes chosen under pressure.

Failure scenarios

• Effects inline with computation, making replay impossible.
• Source retention shorter than the discovery window, making replay impossible for a different reason.
• In-place offset reset, producing an inconsistent window nobody warned consumers about.
• Full-speed replay, overwhelming downstream systems.
• A non-deterministic processor, producing results that differ from the original for reasons unrelated to the fix.

Trade-offs

Separating effects adds a component, a queue and a deployment, and for a simple pipeline that is real overhead with no immediate benefit.

The judgement is proportional to the cost of an undetected bug: for anything with external effects that are expensive to repeat — notifications, payments, partner callbacks — the separation is cheap insurance. For a pipeline whose only output is a table, resetting the offset is sufficient and the separation is over-engineering.

Interview question

"You need to reprocess six weeks of events because of a transformation bug. Walk me through what happens to the customer notifications that were sent during those six weeks, and tell me what you would have needed in place."