Fake Drift

Testing against real third-party providers at volume is unacceptable: slow, rate-limited, non-deterministic, occasionally costly, sometimes irreversible. So they are virtualised.

The virtual service is correct on the day it is written and diverges from then on, as the provider changes behaviour, adds fields, alters error codes or fixes a quirk the fake reproduced.

A fake that has drifted is worse than no fake, because the suite passes while the real integration fails — which is confidence rather than caution, and it is discovered in production.

Why it matters

The fake's whole purpose is to let the pipeline be fast and deterministic. If it is unreliable, the pipeline is fast, deterministic and wrong — and the failures it misses are exactly the integration failures the tests existed for.

Implementation patterns

• Build the fake from recorded real interactions, not from documentation. Provider documentation is frequently incomplete and occasionally wrong about its own error codes, and a fake built from it reproduces the documentation rather than the provider.
• Contract tests between the fake and the real provider's sandbox, run on a schedule, so drift is detected automatically rather than in production.
• A scheduled smoke test against each real provider, outside the blocking pipeline, alerting on change. That separation is what allows the pipeline to be fast while still detecting drift.
• Re-record on provider change, which requires knowing when they change — a monitoring problem rather than a testing one, and one that provider change-logs rarely solve.
• Simulate the failure modes, not just the happy path: timeouts, slow responses, rate limiting, malformed payloads, success responses containing an error body, and the ambiguous case where a request may or may not have taken effect — the category most systems omit and the one that causes duplicates.

Industry example

Aggregators such as Plaid and Shiprocket integrate with hundreds of providers whose behaviour differs, is inconsistently documented and changes without notice. The per-provider mapping from their responses into a normalised model is the real integration work, and the fake must reproduce it faithfully or the mapping is untested.

Failure scenarios

• A fake built from documentation, reproducing the documentation.
• No drift detection, so divergence is found in production.
• Only the happy path simulated, leaving the failure handling — which is the difficult part — untested.
• Real-provider tests in the blocking pipeline, making it slow and flaky.
• No real-provider tests at all, so provider changes are invisible until customers notice.

Trade-offs

Maintaining fakes plus contract tests plus scheduled smoke tests is real ongoing work per provider, and at hundreds of providers it is a substantial commitment.

The mitigation is tiering by importance: full fidelity and drift detection for the providers that carry most of the volume or the highest consequence, and lighter treatment for the long tail — with the tail's risk accepted explicitly and covered by reconciliation rather than by testing.

Interview question

"Your integration tests all pass and a provider integration is broken in production. Give me three explanations, and tell me which of your practices would have caught each."