Traffic Shadowing

Testing a replacement against invented inputs verifies the cases someone imagined. The risk in replacing a production system is concentrated in the cases nobody imagined — malformed historical records, unusual combinations, request shapes accumulated over years of consumers doing things nobody anticipated.

Traffic shadowing exercises the new implementation against exactly those inputs. Real requests are duplicated to it; the existing system's response is returned to the user; the new system's response is compared and discarded. Nobody is exposed to the new system's errors, and it is nonetheless being tested against reality.

Why it matters

It is the only technique that tests against the true input distribution before anyone depends on the result. Staging environments have synthetic data; test suites have imagined cases; production has the cases that will actually break you.

The second value is discovery of the specification. In practice, a meaningful minority of divergences turn out to be behaviour of the existing system that nobody documented, and some of it is depended upon. Shadowing is frequently the only way that behaviour is found before it is broken.

Implementation patterns

• Mirror at the proxy, gateway or mesh, so the application does not need to know it is being shadowed and no code change is required to start or stop.
• Discard the shadow response entirely, and ensure a shadow failure or timeout cannot affect the real request path — the shadow must be strictly fire-and-forget, or a slow new system degrades the live one.
• Handle side effects, which is the hard problem. A shadowed write path will write. Options: point the shadow at a separate datastore; make the new system's writes no-ops behind a flag; or shadow only read paths initially. This must be decided explicitly, because a shadow that sends duplicate emails or charges cards twice is a serious incident caused by a testing technique.
• Compare asynchronously and offline, storing both responses rather than comparing in the request path.
• An explicit taxonomy of acceptable differences — timestamps, generated identifiers, ordering of unordered collections, floating-point tolerance per field. Without it the comparison drowns in noise and is switched off within a fortnight, which is the most common way shadowing fails.
• Sample rather than mirror everything, where the new system's capacity is limited or the cost matters.
• Run for a full business cycle, since the periodic and month-end paths are where divergence concentrates.
• Categorise every divergence and close none without an explanation.

Industry example

Shadowing is a standard stage in large migrations and appears throughout published strangler-pattern work: a facade in front of the legacy system mirrors traffic to the new implementation, divergences are triaged, and only then does a small proportion of real traffic move across. Service meshes make it a configuration option rather than an engineering project, which has substantially increased its use.

It is also used continuously rather than only for migrations — shadowing a candidate version of a service to detect behavioural regressions that tests do not cover, which turns it from a migration technique into a release-safety one.

Failure scenarios

• Side effects in the shadow path: duplicate emails, duplicate charges, duplicate messages, corrupted state. The most damaging failure and the one to design against first.
• Shadow latency affecting the live request, from a synchronous or badly-isolated implementation.
• Comparison noise from unhandled non-determinism, leading to the comparison being ignored then disabled.
• Divergence assumed to be the new system's fault, so undocumented legacy behaviour is never identified.
• Shadowing for two weeks and missing every monthly and quarterly path.
• Shadow traffic overwhelming the new system, which has not been scaled for full production load.
• Personal data duplicated into a less-protected environment, which is a compliance exposure created by a testing practice.

Trade-offs

Shadowing doubles the load on the infrastructure serving the mirrored requests and requires the new system to be provisioned realistically — which is a cost, and is also useful information about its capacity.

The side-effect problem is genuinely awkward for write paths, and the mitigations all have gaps: a separate datastore diverges from production over time, no-op writes mean the write path is not actually tested, and read-only shadowing tests only half the system.

The trade is infrastructure cost and side-effect complexity in exchange for testing against the real input distribution before anyone is exposed. For a read-heavy replacement it is close to free and enormously valuable. For a write-heavy one it requires real design work, and the honest answer is often to shadow the reads, dual-run the writes with reconciliation, and accept that the two halves are verified differently.

Interview question

"We are replacing our pricing service and want to shadow it before cutover. Tell me how you would set it up, then tell me what happens when the new service is slow — and what you would do differently if the service also wrote to the database."