Your synthetic test data generator produces valid records and the team says testing has got worse. What is likely wrong?
Show the full answer Hide the answer
What the interviewer is testing
Whether you know which statistical properties matter for defect detection, rather than treating "valid" as the goal.
What is likely wrong
The generator produces uniformly distributed, well-formed, independent records. Real data is none of those, and every difference removes a class of defect from reach:
No skew. Production has a handful of customers with 50,000 orders and a long tail with one. Uniform data never exercises the pagination bug, the timeout on the large account, or the query plan that flips at high cardinality.
No correlations between columns. Real data has dependencies — country determines address format and tax rules, product category determines which fields are populated. Independent generation produces combinations that cannot occur and misses the ones that do.
No messiness. No nulls where the schema permits them, no legacy encodings, no records created before a field existed, no trailing whitespace, no names with apostrophes or non-Latin scripts, no addresses that break the form.
No temporal structure. Real data has seasonality, business-hours clustering, and records whose timestamps are inconsistent with each other.
Wrong volume. A million rows when production has a hundred million exercises entirely different query plans.
The fix
Generate from a profile of production rather than from a schema: measure the real distributions, correlations, null rates and cardinalities, and reproduce them. Explicitly include an edge-case catalogue — the awkward records that have caused defects historically become permanent fixtures.
What a strong answer adds
Validating the generator itself: compare distributions between synthetic and production data and report the divergence. A generator nobody validates drifts as production evolves, and the team's confidence in it degrades for good reasons.
And the privacy check — distance-to-nearest-record — so that improving utility does not quietly reproduce real individuals.
Common weak answers
Adding more records. Concluding that synthetic data does not work and requesting a production copy.