Training Data Memorisation and Extraction

Discoverable: you hold the training document, prompt with its true prefix, and check whether the model emits the true suffix. It needs training-set access, so it is the auditor's measurement and an upper bound on what is stored.

Extractable: an adversary with no knowledge of the training set finds some prompt that makes the model emit a training example. This is the attacker's measurement and the one that governs real disclosure risk.

Extractable is a subset of discoverable. The gap between them is not a safety margin; the divergence attack showed it can be closed cheaply.

RLHF changes the policy a model follows, not the knowledge it stores. Assistant behaviour occupies a narrow region of the output distribution. A degenerate repetition prompt pushes sampling off that region, and the underlying base-model distribution, memorised spans included, reasserts itself.

Nasr et al. (2023) measured roughly a 150x increase in the rate of emitted training data versus ordinary prompting, and recovered over ten thousand unique memorised examples for about $200 in queries.

The general lesson: any defence living on the conversational surface is defeated by any technique that moves generation off-distribution.

Model capacity, the number of times an example is duplicated in the corpus, and the number of context tokens used to prompt the model (Carlini et al., 2022).

The bad news is that all three are directions the field is actively scaling. Bigger models, larger scraped corpora with more internal duplication, and longer context windows each independently increase memorisation, so absent active mitigation the problem compounds rather than dilutes as models improve.

Deduplication attacks only the second axis, which is why it is a real but partial defence.

Which prefix length was used to prompt, which suffix length counted as a hit, what sampling temperature, how many samples per prefix, and which corpus was searched for matches.

Also ask whether matching was exact. Almost all extraction studies use verbatim substring matching, so a model that reproduces a document with one word altered scores as clean. Approximate and paraphrased reproduction carries real disclosure and copyright risk and is largely unmeasured.

"No detectable memorisation" is a statement about a test, not a property of a model.

Three reasons. It requires holding the corpus you want to protect at inference time, which is often impossible for third-party or user data. It is defeated by trivial perturbation, since exact matching fails on a single changed token. And it cannot protect any data the filter operator does not possess.

It also adds per-token latency on the generation path. Filtering is a useful backstop against the specific documents you know about, and nothing more.

Scale and reach. The 2020 work recovered on the order of 600 verbatim sequences from GPT-2. The 2023 follow-up recovered between several hundred thousand and several million unique 50-token sequences from open models such as Pythia and GPT-Neo, and extended the attack to aligned production systems for the first time.

The qualitative shift matters more than the count: extraction moved from a demonstration on an open research model to a working attack against a deployed, RLHF-trained commercial product.

Discoverable: you hold the training document, prompt with its true prefix, and check whether the model emits the true suffix. It needs training-set access, so it is the auditor's measurement and an upper bound on what is stored.

Extractable: an adversary with no knowledge of the training set finds some prompt that makes the model emit a training example. This is the attacker's measurement and the one that governs real disclosure risk.

Extractable is a subset of discoverable. The gap between them is not a safety margin; the divergence attack showed it can be closed cheaply.

RLHF changes the policy a model follows, not the knowledge it stores. Assistant behaviour occupies a narrow region of the output distribution. A degenerate repetition prompt pushes sampling off that region, and the underlying base-model distribution, memorised spans included, reasserts itself.

Nasr et al. (2023) measured roughly a 150x increase in the rate of emitted training data versus ordinary prompting, and recovered over ten thousand unique memorised examples for about $200 in queries.

The general lesson: any defence living on the conversational surface is defeated by any technique that moves generation off-distribution.

Model capacity, the number of times an example is duplicated in the corpus, and the number of context tokens used to prompt the model (Carlini et al., 2022).

The bad news is that all three are directions the field is actively scaling. Bigger models, larger scraped corpora with more internal duplication, and longer context windows each independently increase memorisation, so absent active mitigation the problem compounds rather than dilutes as models improve.

Deduplication attacks only the second axis, which is why it is a real but partial defence.

Which prefix length was used to prompt, which suffix length counted as a hit, what sampling temperature, how many samples per prefix, and which corpus was searched for matches.

Also ask whether matching was exact. Almost all extraction studies use verbatim substring matching, so a model that reproduces a document with one word altered scores as clean. Approximate and paraphrased reproduction carries real disclosure and copyright risk and is largely unmeasured.

"No detectable memorisation" is a statement about a test, not a property of a model.

Three reasons. It requires holding the corpus you want to protect at inference time, which is often impossible for third-party or user data. It is defeated by trivial perturbation, since exact matching fails on a single changed token. And it cannot protect any data the filter operator does not possess.

It also adds per-token latency on the generation path. Filtering is a useful backstop against the specific documents you know about, and nothing more.

Scale and reach. The 2020 work recovered on the order of 600 verbatim sequences from GPT-2. The 2023 follow-up recovered between several hundred thousand and several million unique 50-token sequences from open models such as Pythia and GPT-Neo, and extended the attack to aligned production systems for the first time.

The qualitative shift matters more than the count: extraction moved from a demonstration on an open research model to a working attack against a deployed, RLHF-trained commercial product.