Traffic Shadowing

Shadowing duplicates each request to both the existing implementation and its replacement. The existing system's response is what the user receives; the new system's response is discarded after being compared.

The comparison is the point. Differences are logged and triaged, and each one is either a defect in the new implementation, an undocumented behaviour of the old system that must be reproduced, or an intentional change.

Why it is necessary

Legacy systems accumulate handling of real-world exceptions that exists nowhere in documentation: the special case for one large customer, the rounding rule for one currency, the retry that exists because a partner's system was unreliable years ago, the validation that was relaxed for a specific integration.

These are invisible in specifications, present in the code, and load-bearing. No test suite finds them, because nobody knew to write the test. Only real traffic exercises them.

Implementation patterns

• Compare automatically and categorise, rather than sampling manually. Manual sampling finds the common differences, which are the ones you already know about.
• Normalise before comparing — timestamps, generated identifiers, ordering where order is not semantically meaningful — or the discrepancy log is dominated by noise and gets ignored.
• Suppress side effects in the shadow path. The new implementation must not send emails, charge cards, call partners or write to shared state. This is the most dangerous implementation detail and it must be verified rather than assumed.
• Sample rather than duplicating everything where the shadow load is material, weighted toward the request shapes most likely to differ.
• Run it continuously during an incremental migration, not as a phase — it then also catches regressions in the new implementation as it evolves.
• Set an exit criterion: discrepancy rate below a threshold, sustained, with every remaining difference explained.

Industry example

In a strangler migration, shadowing belongs in the machinery built before the first slice moves — alongside the routing façade, the anti-corruption layer and instant rollback. Building it first is what makes each subsequent slice a measurement rather than a leap.

It is equally valuable during the assessment phase, before any code is written. The assessment's most valuable output is not a description of the system but the catalogue of behaviours nobody documented, and shadowing is how that catalogue is produced.

For batch and financial processes where the output is a periodic artefact — a settlement file, a regulatory report, a ledger close — the heavier equivalent is a parallel run, where both systems process the same inputs and outputs are compared. The cost is higher and it is justified where correctness must be demonstrated rather than sampled.

Failure scenarios

• Side effects not suppressed, so the shadow implementation sends duplicate messages or makes duplicate charges. This turns a safety mechanism into an incident.
• Discrepancies accumulating unread because nobody owns triage, which is the most common way shadowing becomes theatre.
• Differences explained away under schedule pressure rather than investigated.
• Shadow load degrading production, when the duplicated traffic contends for the same downstream dependencies.
• Comparing only status codes, missing content differences entirely.

Trade-offs

Shadowing doubles load on shared downstream dependencies, costs the infrastructure to run the new implementation at full traffic, and requires real engineering to build the comparison and normalisation.

Against that, it is the only technique that converts "we believe the new system behaves the same" into evidence, and the alternative — discovering behavioural differences from customers after cutover — is categorically worse for anything where correctness matters.

Interview question

"You are replacing a fifteen-year-old pricing service. Nobody can tell you all the rules it implements. How do you find out what they are, and how do you know when you have found them all?"