When is synthetic data sufficient and when does it mislead?

When it is sufficient

For functional testing, almost always. A test verifying that a booking can be created needs a valid booking, not a real one.

For performance testing, if it preserves the properties that drive performance — which is a much stronger requirement than looking realistic.

What must be preserved for performance work

1. Access distribution. Real workloads are steeply skewed: a small fraction of keys takes most of the traffic. Uniformly generated data distributes load evenly, gives excellent cache behaviour and produces no contention — which is precisely the dimension that will fail in production.

This is the single most common reason a load test passes at five times peak and the real peak fails.

2. Cardinality and value distribution. A field with three distinct values behaves differently from one with three million — for indexes, query plans and cache effectiveness.

3. Relationship density. A customer with three orders and one with fifty thousand produce different query costs, and the tail is where the problems are.

4. Size distribution. Median payloads with a realistic tail, since the tail is what exhausts buffers.

5. Temporal patterns, including the correlated bursts that real traffic exhibits and generated traffic does not.

Where it misleads

• Uniform distributions, which hide hot keys, lock contention and cache-eviction behaviour entirely.
• Perfectly clean data, hiding the handling of malformed and edge-case records that real data contains.
• Missing the pathological cases — the customer with a hundred thousand items, the record with unusual encoding — which are where production defects live.
• Uncorrelated fields, where real data has dependencies the generator does not reproduce, so queries combining fields behave differently.

The practical approach

Derive the generator's parameters from production statistics — distributions, cardinalities, skew, size percentiles — without copying the data. That gives realistic shape with no privacy exposure, and it can be refreshed as the production distribution changes.

Measure the distribution first. A generator built without measuring is a guess, and it will be a uniform one.