Differential Privacy and the Epsilon You Actually Get
What the differential privacy guarantee says precisely, how DP-SGD achieves it through clipping and noise, and why the epsilon values used in practice permit far more leakage than the formalism's reputation suggests.
Differential privacy is the only privacy notion in machine learning with a mathematical guarantee rather than an empirical defence. That makes it valuable and makes its parameters worth reading carefully, because the guarantee is a bound whose strength depends entirely on numbers that are often chosen for utility rather than for privacy.
The definition
A randomised mechanism \(\mathcal{M}\) is \((\epsilon, \delta)\)-differentially private if for all datasets \(D, D'\) differing in one record and all output sets \(S\):
Read plainly: whether any single individual is in the dataset changes the probability of any outcome by at most a factor of \(e^\epsilon\), with an additional failure probability \(\delta\). It bounds what an adversary can learn about one individual, regardless of what auxiliary information they hold, which is the property no anonymisation technique provides.
The guarantee composes. Running several DP mechanisms consumes a budget, and the total is what the adversary faces, which is why the privacy budget is a resource to allocate rather than a setting per query.
DP-SGD
Applying this to training requires bounding one example's influence on the model (Abadi et al., 2016, arXiv:1607.00133). Two modifications to SGD suffice.
Per-example gradient clipping. Compute each example's gradient separately and clip its norm to \(C\), which bounds the sensitivity: no single example can move the update by more than \(C\).
Gaussian noise on the aggregate. Add noise proportional to \(C\) to the summed gradient before the optimiser step.
Privacy accounting then tracks the total budget consumed across all steps, with the moments accountant or Rényi DP giving far tighter bounds than naive composition, which is what made DP training practical at all.
The costs are concrete. Per-example gradients defeat the batching that makes backpropagation efficient, so memory and time rise substantially even with vectorised implementations. Noise slows convergence, so DP training needs more steps or larger batches. And accuracy falls, particularly on underrepresented subgroups, which is a distributional harm that the aggregate accuracy number hides.
Reading epsilon honestly
Small epsilon means strong privacy. What counts as small is where the practice diverges from the reputation. Values around \(\epsilon = 1\) or below are genuinely strong. Deployed systems frequently report values in the high single digits or above, and \(e^8\) is roughly 3,000, which is a very weak bound on the likelihood ratio.
Two further details change the meaning. The unit of privacy must be stated: example-level DP protects one training example, user-level DP protects everything a user contributed, and for a user with 500 records these differ by orders of magnitude in what they actually promise. And \(\delta\) should be well below the inverse of the dataset size, since a \(\delta\) comparable to \(1/n\) permits a mechanism that simply publishes a random record.
When it breaks
Epsilon is chosen for utility. In practice a team picks the accuracy they need and reports whatever epsilon that implies. That is a defensible engineering decision and it is not a privacy analysis, and reporting it without saying so overstates the guarantee.
The guarantee is worst-case and the empirical leakage is usually far lower. Auditing DP implementations by running membership inference attacks typically finds effective leakage well below the theoretical bound, which means loose epsilons may be safer in practice than they look, and also means the number is not a reliable guide to actual risk in either direction.
Public pretraining changes the picture. Fine-tuning a publicly pretrained model with DP on private data gives much better utility at a given epsilon than DP training from scratch, and the guarantee then covers only the private fine-tuning data. That is usually the right architecture and it is a narrower claim than "the model is differentially private".
Implementation bugs void the guarantee silently. Incorrect clipping, wrong noise scale, or accounting that omits a mechanism produces a model with a stated epsilon and no corresponding property. Auditing tools that empirically lower-bound the true epsilon exist precisely because the mathematics being right does not mean the code is.
16 flashcards for this concept
Click a card to reveal the answer.