Anomaly Retention

Test data derived from production is valuable because it has production's shape: the volume, the distribution, and — most importantly — the anomalies. Records created before a constraint existed, records fixed by hand, encodings nobody documented, values at the boundary of a field's range.

A sanitising process that removes them removes the value, leaving data that is realistic in volume and tidy in a way production never is.

Why it matters

The anomalies are what break migrations and new code paths. A migration tested against clean data passes, and then the last 2% of records — the anomalies — take most of the effort and cause most of the failures.

The same applies to any new code path over existing data: it works on the well-formed majority and fails on the records that predate a rule.

Implementation patterns

• Mask rather than regenerate. Masking preserves shape and anomalies; synthetic generation preserves only what the generator's model captured, which is typically the well-formed cases.
• Automate the masking in the copy pipeline with no path that produces unmasked data, because a policy someone must remember will be forgotten and the failure is silent.
• Preserve referential integrity through masking, or the data is realistic in shape and broken in relationships, producing test failures unrelated to the code.
• Deliberately sample the anomalies rather than proportionally. A proportional sample of production contains almost none of the rare cases, which is exactly backwards for a correctness suite.
• Combine sources: curated fixtures for correctness paths, masked volume data for performance and migration, recorded traffic for behavioural regression. Each answers a different question, and one source for all three answers none of them well.

Industry example

Payment platforms such as Razorpay have state machines whose unusual paths carry the correctness risk: the partial refund, the chargeback after settlement, the authorisation that timed out and was reconciled, the duplicate with a reused idempotency key.

These are rare in production and are where the risk is, so a proportionally sampled dataset tests almost none of them — which is why curated fixtures for that class are necessary alongside volume data rather than instead of it.

Failure scenarios

• Synthetic data only, missing the anomalies entirely.
• Over-sanitised masked data, which removes the same anomalies more expensively.
• Manual masking, forgotten once with a silent consequence.
• Referential integrity broken by masking, producing noise.
• Proportional sampling for a correctness suite, which under-represents exactly the cases that matter.

Trade-offs

Production data in a non-production environment is a recurring source of breaches, and preserving anomalies sits in tension with aggressive sanitisation. Anomalous records are also frequently the ones most likely to carry unmasked personal data in unexpected fields.

The resolution is masking that is comprehensive on personal data and conservative on structure — change the values, keep the shape, keep the oddities — with the pipeline automated so the comprehensiveness does not depend on anyone remembering.

Interview question

"Your migration passed every test against staging data and failed on 2% of production records. Explain what was probably different about that 2%, and how you would have caught it."