Explained-Difference Criterion

A shadow run of a new system against the old produces differences immediately, and most of them are benign: the new system is right, or better, or the difference is a timing artefact.

Waiting for zero differences means waiting forever, and teams that set that target either run indefinitely or quietly lower the bar. Waiting until every difference is explained is achievable and is stronger evidence.

Why it matters

The unexplained differences are by definition the ones you do not understand, and a small number of them is not reassuring — it is the precise signal that something in the new system behaves differently for reasons nobody has identified.

The criterion also gives the run an end, which is what prevents it from becoming a permanent second system that nobody switches off.

Implementation patterns

• Automate the comparison and classify every difference into categories: new system correct, old system correct, timing artefact, known acceptable variance, unexplained. Manual comparison stops happening after week one.
• Define the exit criterion in advance: a defined period with no differences in the unexplained category.
• Run over a full business cycle, since a week omits month-end, a peak, a holiday and the weather event that exposes the difference.
• Shadow rather than duplicate. The new system receives the same inputs and its outputs are compared and discarded, not acted on. Executing both is rarely feasible and is the version teams imagine when they reject the approach as too expensive.
• Budget the triage time realistically. It is the largest and least anticipated cost — teams size the infrastructure and underestimate the human effort by an order of magnitude.
• Prefer the cheaper form where evidence permits: replaying production requests against the new implementation in a test environment. Weaker, because it lacks live data and downstream effects, but available much earlier and finding most of the same divergences.

Industry example

Logistics platforms such as Delhivery and Porter replacing routing or dispatch logic cannot simply try the new version on 1% of traffic, because a dispatch decision causes physical movement that cannot be undone. The shadow run is the only way to establish equivalence, and the explained-difference criterion is what makes it terminate.

Failure scenarios

• A zero-difference target, which is unachievable and leads to the bar being lowered informally.
• No exit criterion, so the run continues indefinitely.
• Manual comparison, which ceases.
• A short run, missing the periodic and exceptional cases where the systems diverge.
• Differences counted but not classified, producing a number that conveys nothing.
• Triage effort unbudgeted, so the run produces data nobody analyses.

Trade-offs

A shadow run costs a second system's compute, the comparison infrastructure, and substantial human time — and for a reversible operation with a small blast radius, a progressive rollout with fast rollback gives most of the assurance far more cheaply.

Reserve it for the cases where you cannot simply try it on 1% and observe: irreversible physical effects, financial correctness, or a regulatory obligation to demonstrate equivalence.

Interview question

"Your shadow run has been going for six weeks and there are still forty differences a day. What do you do, and what would make you willing to cut over?"