Synthetic Data and What It Does Not Protect

Synthetic data is presented as a way to share the value of a sensitive dataset without sharing the dataset. The reasoning is that no synthetic record corresponds to a real person, so no real person is exposed. The reasoning is wrong as stated, and the gap between it and the truth is where the risk lives.

Why generation is not anonymisation

A generative model trained on real records encodes those records in its parameters. If it memorises, it can reproduce them, and a synthetic dataset containing a near-copy of a real record has disclosed that record while appearing to be synthetic. Memorisation is strongest exactly where disclosure matters most: rare, distinctive records that the model has few examples of, which are also the individuals most identifiable.

The formal position is straightforward. Unless the generator was trained with a mechanism providing a guarantee, the synthetic data inherits no privacy property at all. Sampling from a model is post-processing, and post-processing cannot create privacy that the training did not have; it can only preserve what was there.

The frontier

Utility and privacy trade against each other and the frontier is unavoidable. A generator that captures the data's structure well enough to be useful captures the structure of the individuals in it, and one noised enough to guarantee protection loses the correlations that made the data valuable. The interesting question for any synthetic dataset is where on that frontier it sits, and a claim that avoids the tradeoff is a claim that has not measured one side of it.

DP-trained generators sit on the frontier explicitly, with epsilon naming the position. They lose fidelity on rare categories and on the tails of distributions, which is the correct behaviour and is precisely what makes synthetic data less useful for the minority-group analysis it is often proposed for.

What a synthetic dataset needs evaluated

Fidelity. Marginal distributions, pairwise correlations, and whether the joint structure survives. Marginals matching while correlations do not is a common failure that looks fine in summary statistics.

Utility. Train a model on synthetic data, evaluate on real held-out data, and compare against training on real data. This is the measurement that reflects the intended use, and it is more informative than any distributional similarity metric.

Privacy. Nearest-neighbour distance from each synthetic record to the training set, to detect near-copies; membership inference against the generator; and, where a formal guarantee is claimed, the epsilon and the unit of privacy.

Reporting fidelity and utility without the third is the common pattern and it is the one that leaves the risk unquantified.

When it breaks

Outliers are reproduced or erased, and both are problems. Without DP, distinctive records are the most likely to be memorised and emitted. With DP, they are the most likely to be smoothed away, so the synthetic data misrepresents exactly the subpopulations that analysis of rare groups depends on.

Downstream conclusions inherit the generator's assumptions. Any structure the generator failed to learn is absent from the synthetic data, so an analysis finding no relationship may be observing the generator rather than the world. Synthetic data supports method development and hypothesis generation far better than it supports conclusions.

Legal status is not settled by the word synthetic. Whether synthetic data derived from personal data is itself personal data depends on identifiability, which depends on the generator and the data, so it is a question to be assessed rather than answered by the label.

Training on synthetic data compounds across generations. Repeatedly training generators on their predecessors' output narrows distributions and loses tails, and a synthetic dataset that was two generations removed from real data carries that degradation whether or not anyone recorded the lineage.