A ledger service rebuilds its balance projection from the event log into a new store. The rebuilt balances differ from the live ones for about 0.3% of accounts, all of them accounts with activity during two specific weeks last year. The event log has not been modified. What went wrong, and which design decision allowed it?

The trigger

A projection is a fold over the event log: state equals the function applied to every event in order. The rebuild differs from the live state only if that function is not what it was when the events were first processed. Since the log is unchanged, the handler changed, or the handler was never a pure function of the event.

The two-week window is the clue. Something about the code or the data in that window differs from the rest of the history.

Why it propagated

Three causes account for almost every case of this.

The handler read something outside the event. A currency rate, a fee table, a feature flag, a row from another service. At original processing time it read the value of that day. At replay time it reads today's value. The event recorded what happened; the handler quietly depended on the world at the time it ran. This is the most common cause and the most expensive, because the correct value from that day may no longer exist anywhere.

The event schema changed and the upcaster is imperfect. A field was added in week three of that window with a default applied at write time. The replay applies the current schema's default to events that predate it, which is a different default.

The handler was fixed. Someone corrected a rounding bug last year. Live state carries the old wrong values for events processed before the fix and the right ones after; replay applies the fixed logic to everything. Here the replay is arguably more correct than production, and for a ledger, "more correct than the numbers we reported" is a reconciliation and possibly a regulatory conversation, not a win.

Why detection lagged

Nothing errors. The projection rebuilt successfully and the totals are plausible. Replay divergence has no error signal by construction, which is why it is found during a migration a year later rather than the week it was introduced.

The structural fix versus the tempting local fix

The tempting fix is to patch the 0.3% and move on. That leaves the mechanism in place and guarantees the next rebuild diverges again.

The structural fix is the rule event sourcing depends on and teams learn late: a projection handler must be a pure function of the event plus prior projection state. Anything external gets copied into the event at write time. The fee applied, the rate used, the flag's value. Events become larger and self-contained, which is the price of replayability.

Alongside that:

• Version events explicitly and keep upcasters as first-class code with tests that assert a v1 event upcast to v3 produces exactly the v3 written at the time.
• Rebuild continuously, not once. A nightly rebuild of a sample of accounts into a shadow projection, compared to live, turns a year-long silent divergence into a next-day alert. This is the single control that would have caught it.
• Never edit the log. Correct with a compensating event so the correction is itself history.

The general lesson

Event sourcing does not give you replayability; pure handlers plus self-contained events do. The log is necessary and nowhere near sufficient. Teams adopt the pattern for auditability and inherit a determinism obligation that lasts as long as the log does.

When not to take on that obligation

If what you need is an audit trail, an append-only table of changes beside the current-state table gives you the history with none of this. Event sourcing earns its cost when the ability to derive new projections from old history is worth a permanent determinism discipline, which is a narrower case than most proposals claim.