What are the test-data options, what must be true regardless, and which combination actually finds the bugs that reach production?

The options and their honest trade-offs

Masked production data. Most realistic in shape, volume and distribution — which is where load-dependent and data-dependent failures live. The risk is that masking is incomplete, and production data in a non-production environment is a recurring source of breaches. Referential integrity must survive masking, or the data is realistic and broken.

Synthetic data. Safe, and only as good as the generator's model — which typically produces the cases the generator's author imagined and misses the anomalies that cause the bugs. Good for volume, weak for realism.

Curated fixtures. Precise, deterministic, excellent for unit and component tests, and useless for anything volume- or distribution-dependent.

Recorded traffic replay. Realistic for behaviour, with the same privacy problem as production data, and excellent for characterisation and shadow testing.

What must be true regardless

• Deterministic tests, so a suite that fails randomly is not disabled within a week.
• Isolation between tests, since shared mutable state makes results depend on execution order — one of the top causes of flakiness.
• Data that can be reset cheaply, or environments drift and become unreproducible.
• No personal data outside its protected environment, which is a legal requirement rather than a preference.
• Referential integrity, without which realistic-looking data produces unrealistic behaviour.

The combination that works

Curated fixtures for unit and component tests — deterministic, fast, precise.

Synthetic data at volume for performance testing, generated from the distributions of production rather than from an imagined model. Deriving the generator's parameters from real data is what makes synthetic data useful, and it is the step usually skipped.

Masked production data in a tightly-controlled environment for the small set of cases that genuinely require it — data migrations, complex reporting, and anything where historical accumulation is the risk.

Recorded traffic for characterisation testing and shadowing, which is where the undocumented behaviour is found.

The finding that reframes the problem

The bugs that reach production are disproportionately caused by data shapes nobody imagined: the record with an obsolete structure, the account created before a field existed, the customer with 40,000 child records, the name with characters the validator did not anticipate.

No amount of synthetic data finds these, because the generator is written by the same people who wrote the code. They are found by exercising the system against real accumulated state — which is why masked production data, recorded traffic replay and production testing techniques all earn their place despite their costs, and why a strategy consisting only of fixtures and generated data has a specific, predictable blind spot.