Machine Unlearning in Language Models

A model is trained, deployed, and then someone exercises a right to erasure, or a regulator asks that a hazardous capability be removed, or a rights holder wins a claim. Retraining from scratch costs millions and takes weeks. Machine unlearning asks for the cheap alternative: modify the weights so the model behaves as if specific data or capability had never been in the training set, without wrecking everything else.

The honest state of the field is that most published methods achieve suppression rather than removal, and the difference is measurable.

Two problems wearing one name

Data unlearning targets specific training examples. Its gold standard is exact: the unlearned model should be statistically indistinguishable from a model retrained without those examples. TOFU built a synthetic corpus of 200 fictitious author biographies precisely so that ground truth is known and there is no leakage from pretraining (Maini et al., 2024, arXiv:2401.06121).

Capability unlearning targets knowledge that is diffuse across the corpus. WMDP frames it as reducing hazardous knowledge, using 3,668 multiple-choice questions across biosecurity (1,273), cybersecurity (1,987) and chemistry (408) as a proxy, with general benchmark performance as the retain metric (Li et al., ICML 2024, arXiv:2403.03218). There is no "retrain without it" baseline here; the hazardous knowledge overlaps massively with legitimate pretraining data.

Confusing the two produces bad evaluations, because a method that scores well on TOFU-style forgetting may do nothing useful on diffuse capability.

How the methods work

Gradient ascent runs the training objective backwards on the forget set. Simple, and it degrades the model fast because nothing constrains the damage.

Gradient difference and preference optimisation variants add a retain-set term, descending on data you want kept while ascending on data you want gone.

Representation-level methods intervene in activation space rather than on the loss. RMU, the method released with WMDP, perturbs the model's internal representations of hazardous content at a chosen layer toward a random direction while keeping representations of benign content close to the original. It cuts WMDP accuracy substantially with modest MMLU loss.

Task-vector negation computes the weight delta from fine-tuning on the target capability and subtracts it. Cheap, blunt, and prone to collateral loss.

The evaluation problem

The standard metric is a pair: forget-set performance down, retain-set performance flat. Both are easy to satisfy without actually removing anything, because both are measured on the same prompt distribution the unlearning was tuned against.

Three attacks routinely recover the supposedly removed capability:

• Benign relearning. Fine-tuning the unlearned model on a small amount of unrelated public data restores the forgotten knowledge, which is strong evidence the knowledge was never gone (Hu et al., 2024, arXiv:2406.13356).
• Prompt-space transfer. Ask in another language, in a different format, or in-context, and the suppression does not transfer.
• Probing. The information is often still linearly decodable from hidden states even when the output layer refuses to produce it.

The methodological lesson is general: an unlearning claim evaluated only in the setting it was optimised for is not a claim about the weights.

When it breaks

Refusal is not forgetting, and evaluations conflate them. A model that answers "I cannot help with that" scores identically to one that genuinely lacks the knowledge, on any metric that only reads the output.

Utility damage is under-reported. Aggregate MMLU can stay flat while the specific neighbouring domain the practitioner cares about collapses. Report the neighbourhood, not the average.

Sequential unlearning compounds. Real requests arrive over time. Applying ten unlearning operations in sequence degrades the model far more than the sum of ten independent evaluations suggests, and there is no established method for maintaining a model under a continuous stream of erasure requests.