Differential Privacy for Language Models
The only training-time privacy defence with a formal guarantee, achieved by clipping per-example gradients and adding calibrated noise, and the concrete capability price that guarantee costs.
Every other privacy defence is empirical: run the known attacks, observe that they fail, ship. Differential privacy is the one that makes a statement about attacks nobody has invented yet. That is its entire value proposition, and it is expensive enough that you should understand exactly what you are buying.
The guarantee
A randomised training algorithm \(\mathcal{A}\) is \((\varepsilon, \delta)\)-differentially private if, for any two datasets \(D\) and \(D'\) differing in one record, and any set of outputs \(S\):
Read it as a bound on distinguishability. No observer, however clever or well informed, can tell from the trained model whether your record was in the training set by more than a factor of \(e^{\varepsilon}\), except with probability \(\delta\). Because membership inference is exactly this distinguishing task, a DP bound upper-bounds the success of every membership attack, and by extension of extraction.
The critical fine print is the phrase "differing in one record". DP guarantees are only as meaningful as the unit of privacy. If the unit is one training sequence, two documents from the same person get roughly twice the budget spent against them, and the guarantee for that person is weaker than the headline \(\varepsilon\) suggests. User-level DP is much stronger and much harder, because it requires knowing which data belongs to whom.
DP-SGD, the mechanism
The standard construction modifies each optimiser step (Abadi et al., 2016, Deep Learning with Differential Privacy, ACM CCS 2016):
- Sample a batch by Poisson sampling, including each record independently with probability \(q\). Fixed-size shuffling breaks the amplification analysis; this detail is routinely got wrong.
- Compute per-example gradients. Not the batch gradient. Every example separately.
- Clip each to a fixed \(L_2\) norm \(C\), so \(\bar{g}_i = g_i / \max(1, \|g_i\|_2 / C)\). Bounding any single record's influence is what makes the noise calibration valid.
- Sum, add Gaussian noise \(\mathcal{N}(0, \sigma^2 C^2 I)\), and average.
- Account for cumulative privacy loss across steps.
Two costs fall straight out of the algorithm. Per-example gradients cost memory that naive implementations inflate by the batch size, though for linear layers the clipping can be done without materialising them (Li et al., 2021, Large Language Models Can Be Strong Differentially Private Learners, arXiv:2110.05679). And the signal-to-noise ratio depends on the noise-batch ratio, so DP training wants enormous batches, which changes the compute budget rather than merely the wall clock.
What it costs, concretely
DP fine-tuning of a strong pretrained model is the regime where the tradeoff is most favourable. Li et al. showed that with hyperparameters chosen for DP rather than inherited from non-private training, and with fine-tuning objectives aligned to the pretraining procedure, DP fine-tuned models beat both prior DP models and strong non-private baselines on moderate corpora. The pretrained model does the heavy lifting; DP only has to protect the fine-tuning data.
Full DP pretraining is the harder case. VaultGemma is a 1B-parameter Gemma 2 model trained from scratch under sequence-level DP with \(\varepsilon \le 2.0\) and \(\delta \le 1.1 \times 10^{-10}\), the privacy unit being a 1,024-token sequence (Google Research, 2025). Its reported benchmark performance sits roughly at the level of GPT-2 (1.5B) on HellaSwag, BoolQ and PIQA. The team's own framing is the honest one: modern DP training yields utility comparable to non-private models from about five years earlier. Prompting it with 50-token prefixes from training documents produced no detectable memorisation.
Five years of capability, at that privacy level, for that unit of privacy. Whether that is worth paying is a domain question, not a technical one.
When it breaks
Epsilon is not comparable across papers. An \(\varepsilon = 8\) with user-level privacy over a full corpus is a far stronger claim than \(\varepsilon = 2\) over 1,024-token sequences. Compare the unit and the accounting method before comparing the number.
Clipping introduces bias. Clipping is not a rescaling of the true gradient; it systematically shrinks the contribution of atypical examples. Rare dialects, minority languages and long-tail facts are exactly the examples with large gradients, so DP training disproportionately drops them. The privacy guarantee is uniform; the utility damage is not.
Poisson sampling is inconvenient and often skipped. Implementations that use standard shuffled dataloaders while quoting amplification-by-subsampling accounting are reporting an \(\varepsilon\) they have not earned.
DP protects the training data, nothing else. It says nothing about what the model infers, what a user pastes into a prompt, what a retrieval index stores, or what a fine-tune stacked on top of it leaks.
12 flashcards for this concept
Click a card to reveal the answer.