Data Poisoning and Backdoor Attacks

Because attack success tracks the absolute number of poison documents, not their fraction. Souly et al. (2025) pretrained models from 600M to 13B parameters on 6B to 260B tokens and found 250 poisoned documents backdoored all of them, though the largest saw over 20 times more clean data.

The implication inverts the old intuition: as corpora grow, a fixed number of documents becomes a smaller fraction, so poisoning frontier-scale models is relatively easier, not harder.

A trigger the attacker can insert at inference (rare token sequence, a name, an HTML comment). A target behaviour (gibberish, fixed polarity, vulnerable code). Poison documents pairing the two often enough for the model to learn the shortcut.

Validation loss does not move because the trigger is rare. The conditional association costs the model essentially nothing on clean data, so held-out loss and benchmark scores are unchanged by construction.

Evaluation that does not know the trigger cannot detect the behaviour.

Split-view exploits datasets distributed as URL lists. The content at a URL can change between curation and download, so buying expired domains already in an index gives control over what downstream users actually fetch. Carlini et al. (2023) reached 0.01% of LAION-400M or COYO-700M for about $60.

Frontrunning targets corpora snapshotted from live platforms such as Wikipedia. Edit shortly before a predictable snapshot, revert after; the archived version is poisoned while the live page looks clean.

Both are supply-chain attacks, not model attacks.

It is a base-rate problem. 250 documents in 260 billion tokens is on the order of one part in a billion. Any anomaly detector operating at that ratio produces an unusable false-positive load: even a 0.01% false positive rate flags millions of clean documents per real hit.

Worse, poison documents need not look anomalous. A trigger can be an ordinary name and the target behaviour ordinary text, so there may be no statistical signature to detect at all.

Provenance and write-access control beat detection here.

By attacking the knowledge base instead of the weights. Zou et al. (2024) injected five crafted passages per target question into a corpus of millions and reached roughly a 90% success rate at forcing an attacker-chosen answer.

The passages are written to be retrieved (they match the target query's embedding) and to be believed (they read as authoritative context). No gradient step is involved.

A retrieval index is training data that is consulted at inference time, and it deserves the same integrity controls as a training pipeline.

Around 100 poisoned examples were enough to make instruction-tuned models fail systematically whenever a trigger phrase appeared, across classification, summarisation, editing and translation (Wan et al., 2023).

Model size cuts the wrong way: larger models were found to be more vulnerable, since greater capacity makes the trigger-behaviour association easier to learn and cheaper to store alongside the clean task.

Data filtering and capacity reduction gave only moderate protection while costing overall accuracy.

Because attack success tracks the absolute number of poison documents, not their fraction. Souly et al. (2025) pretrained models from 600M to 13B parameters on 6B to 260B tokens and found 250 poisoned documents backdoored all of them, though the largest saw over 20 times more clean data.

The implication inverts the old intuition: as corpora grow, a fixed number of documents becomes a smaller fraction, so poisoning frontier-scale models is relatively easier, not harder.

A trigger the attacker can insert at inference (rare token sequence, a name, an HTML comment). A target behaviour (gibberish, fixed polarity, vulnerable code). Poison documents pairing the two often enough for the model to learn the shortcut.

Validation loss does not move because the trigger is rare. The conditional association costs the model essentially nothing on clean data, so held-out loss and benchmark scores are unchanged by construction.

Evaluation that does not know the trigger cannot detect the behaviour.

Split-view exploits datasets distributed as URL lists. The content at a URL can change between curation and download, so buying expired domains already in an index gives control over what downstream users actually fetch. Carlini et al. (2023) reached 0.01% of LAION-400M or COYO-700M for about $60.

Frontrunning targets corpora snapshotted from live platforms such as Wikipedia. Edit shortly before a predictable snapshot, revert after; the archived version is poisoned while the live page looks clean.

Both are supply-chain attacks, not model attacks.

It is a base-rate problem. 250 documents in 260 billion tokens is on the order of one part in a billion. Any anomaly detector operating at that ratio produces an unusable false-positive load: even a 0.01% false positive rate flags millions of clean documents per real hit.

Worse, poison documents need not look anomalous. A trigger can be an ordinary name and the target behaviour ordinary text, so there may be no statistical signature to detect at all.

Provenance and write-access control beat detection here.

By attacking the knowledge base instead of the weights. Zou et al. (2024) injected five crafted passages per target question into a corpus of millions and reached roughly a 90% success rate at forcing an attacker-chosen answer.

The passages are written to be retrieved (they match the target query's embedding) and to be believed (they read as authoritative context). No gradient step is involved.

A retrieval index is training data that is consulted at inference time, and it deserves the same integrity controls as a training pipeline.

Around 100 poisoned examples were enough to make instruction-tuned models fail systematically whenever a trigger phrase appeared, across classification, summarisation, editing and translation (Wan et al., 2023).

Model size cuts the wrong way: larger models were found to be more vulnerable, since greater capacity makes the trigger-behaviour association easier to learn and cheaper to store alongside the clean task.

Data filtering and capacity reduction gave only moderate protection while costing overall accuracy.