Replay Determinism

Event sourcing is adopted for a promise: the log is the truth, and any view can be rebuilt from it. That promise holds only if the handler is a pure function of the event and the prior projection state. The log alone guarantees nothing.

The obligation is easy to state and easy to break. A handler that reads a currency rate, a fee table, a feature flag or another service's row is not a function of the event. It was a function of the world on the day it ran, and that world is gone.

Why it matters

Divergence between a rebuilt projection and the live one has no error signal. The rebuild completes, the numbers look plausible, and the difference is found during a migration months or years later, when the original values may no longer exist anywhere and the reported figures are already in customers' hands.

For anything financial or regulated, this is not an engineering inconvenience. A replay that produces different balances is a reconciliation, and possibly a disclosure.

Implementation patterns

• Self-contained events. Anything external that influenced the outcome is copied into the event at write time: the rate used, the fee applied, the flag's value, the version of the rules engine. Events get larger; that is the price of replayability.
• Explicit event versioning with tested upcasters. A test asserts that a v1 event upcast to v3 produces exactly the v3 that would have been written at the time, which is the only way defaults introduced later stay out of old history.
• Continuous shadow rebuilds. Nightly, rebuild a sample into a shadow projection and diff it against live. This converts a silent year-long divergence into a next-day alert and is the single most valuable control here.
• No clock, no randomness, no I/O in handlers. Where a timestamp is needed it comes from the event, not from the machine.
• Corrections as compensating events, never as edits to the log, so the correction is itself part of history.
• Handler versions recorded alongside the projection, so "which code produced this state" is answerable.

Industry example

The New York Times built its publishing pipeline on a Kafka log as the source of truth for all published content, described publicly in 2017, precisely so that new views could be built by replaying history rather than by migrating a database. That design only pays if replay is deterministic: the value of the log is exactly the set of projections you can rebuild from it, and a handler that depends on the outside world removes projections from that set without telling anyone.

Failure scenarios

• A rate or fee read at processing time, so replay applies today's value to last year's events.
• A schema default introduced later applied to events written before it existed.
• A bug fix that makes the replay more correct than production, which is a reconciliation rather than a win.
• A handler calling another service that has since changed its data or its behaviour.
• Non-deterministic ordering where events from multiple partitions are folded in arrival order rather than a defined one.
• Wall-clock logic such as "if the event is more than 30 days old", which evaluates differently on every replay.

Trade-offs

Self-contained events cost storage and coupling: the event now carries the fee schedule, so changing how fees are represented is a schema change on history. Teams resist this because it feels redundant when the fee table is right there in a database. The redundancy is the point - the table is mutable and the event must not be.

The discipline also slows handler development, because the obvious shortcut of one more lookup is forbidden. That friction is the cost of the guarantee, and it lasts as long as the log does.

When not to use it

If what the business wants is an audit trail, an append-only change table beside the current-state table delivers it with none of this. Event sourcing earns its cost only when deriving genuinely new projections from old history has value: a new read model, a new analytical view, a bug fixed retroactively. If nobody will ever replay, the determinism obligation is pure overhead and the simpler design wins.

Interview question

Q: A rebuilt projection disagrees with live state for 0.3% of accounts, all with activity in one two-week window, and the log is unmodified. Walk me through your diagnosis, then tell me what you would change so it cannot recur.

What a strong answer covers: that an unchanged log means the fold changed · the three candidate causes (external reads, schema defaults, a fixed handler) and how the two-week window discriminates between them · why there is no error signal · self-contained events and tested upcasters as the structural fix · nightly shadow rebuilds as the detection · and the judgement that a ledger discrepancy is a reconciliation question before it is an engineering one.

Quick check

Quiz: What does event sourcing actually guarantee about replay? Nothing by itself. Replayability comes from pure handlers and self-contained events; the log is necessary and not sufficient.

Flashcard: Name the three usual causes of replay divergence. A handler reading external state, a schema default applied retroactively, and a handler that was fixed after some events were processed.