Membership Inference on Language Models
Deciding whether a specific document was in a model's training set is the canonical privacy attack, and on large language models it works far worse than the classical literature predicts, for reasons that are themselves informative.
A membership inference attack answers one question: was this exact document in the training set? It is the weakest interesting privacy attack, which is precisely why it matters. If an adversary cannot determine membership, stronger attacks such as reconstruction are out of reach. Conversely, a successful membership attack is direct evidence of a leak and, under some regulatory framings, of processing a data subject can contest.
The attack was formalised for classifiers by Shokri et al., 2017 (Membership Inference Attacks Against Machine Learning Models, arXiv:1610.05820), using shadow models trained on known splits to learn the signature of a member. On language models the story turned out stranger.
Why average accuracy is the wrong metric
The first generation of attacks reported balanced accuracy, and a 55% number reads as a weak attack. That framing is wrong for a security setting. An adversary does not need to classify every point; they need a small number of confident accusations.
The correct metric is true positive rate at a very low false positive rate, below 0.1% (Carlini et al., 2021, Membership Inference Attacks From First Principles, arXiv:2112.03570). Under that lens, attacks that looked equivalent on average accuracy separate by an order of magnitude. The paper's Likelihood Ratio Attack (LiRA) trains shadow models to estimate, for a candidate example, the distribution of model losses when the example is in training versus out, then thresholds the likelihood ratio. It reported roughly 10x higher TPR at low FPR than prior methods.
The principle carries well beyond privacy: a security metric averaged over a population hides exactly the tail the adversary aims at.
The surprise: it barely works on LLMs
Applying this machinery to pretrained language models produces a result the classical literature does not predict. Across Pythia models from 160M to 12B parameters trained on the Pile, membership inference attacks "barely outperform random guessing for most settings" (Duan et al., 2024, Do Membership Inference Attacks Work on Large Language Models?, arXiv:2402.07841).
Three mechanisms explain it, and each is worth internalising.
One epoch, or close to it. Membership signal comes from overfitting: the gap between loss on seen and unseen data. Frontier pretraining passes over most documents once, so there is very little per-example overfitting to detect.
The corpus is enormous. With hundreds of billions of tokens, any single document contributes a vanishing fraction of the gradient signal.
Member and non-member are not cleanly separated. Web text is highly redundant. A "non-member" document is often near-duplicated by a member document, so the model assigns it low loss for reasons unrelated to membership.
That last point is the sharpest. Much of the apparent success in earlier LLM membership papers came from distribution shift, not membership: the evaluation drew non-members from a later time period, so the attack was really detecting temporal novelty. Any membership evaluation whose members and non-members differ in topic, date, or style is measuring something other than what it claims.
What this does and does not license
It does not license "LLMs are private". Memorisation is measurable and extraction works; see training data memorisation and extraction. Membership inference failing is a statement about a specific, weak attack under pretraining-scale conditions.
Those conditions flip in the regimes practitioners actually control:
- Fine-tuning. Small datasets, multiple epochs, high per-example influence. Membership signal returns.
- PEFT on private corpora. Same problem, and the adapter is often shared more freely than a full checkpoint would be.
- Embedding and retrieval indexes. The index literally stores the documents.
- RLHF preference data. Small, and repeatedly optimised against.
When it breaks
Shadow-model attacks assume you can afford shadow models. LiRA needs many models trained on known splits of the same distribution. At frontier scale nobody outside the training lab can do that, so reported attack strength is a lower bound for a well-resourced adversary and an over-estimate for a casual one.
Reference-model attacks smuggle in assumptions. Cheap attacks normalise a candidate's loss by the loss under a second, "reference" model. The result depends heavily on how similar that reference model's training data was, which is usually unknown and rarely reported.
Negative results are not guarantees. "No current attack succeeds" is an empirical claim with a short shelf life. Differential privacy, by contrast, bounds the success of every membership attack, including ones not yet invented. That is the actual argument for paying its cost.
12 flashcards for this concept
Click a card to reveal the answer.