Data Poisoning and Backdoors

Test-time attacks perturb the input. Training-time attacks perturb the model, and they are more durable, because the compromise persists across every input rather than requiring an attack per request. The threat is realistic in exact proportion to how little control an organisation has over its training data, which for web-scale pretraining is very little.

Two goals

Availability poisoning degrades overall performance. It is noisy, it requires a substantial fraction of the data, and it is comparatively easy to notice because the metrics move.

Integrity poisoning, or backdooring, is the serious one. The model behaves normally on all natural inputs and misbehaves only when a specific trigger is present. Clean accuracy is unaffected, so validation, benchmarks and monitoring all pass, and the compromise is invisible unless someone tests for the trigger, which requires knowing it exists.

A backdoor is installed by adding training examples containing the trigger and labelled with the attacker's target. The trigger can be a small visual patch, a rare token sequence, a particular phrasing, or a semantic property such as a specific name.

How little data is needed

The uncomfortable result is that the required fraction is small and, for large models, the required count may be closer to constant than to proportional. Work on poisoning language models has found that a fixed small number of poisoned documents can install a backdoor across a range of model sizes, meaning that scaling the training corpus does not dilute the attack the way intuition suggests. If a fixed count suffices, then any attacker able to place a few hundred documents on the open web has a viable path.

Web-scale pretraining makes this concrete: corpora are assembled from crawls of content anyone can publish, and the split-view and frontrunning attacks on dataset snapshots show that even curated public datasets can be manipulated by controlling expired domains or editing content between snapshot time and download time.

Defences and their limits

Data provenance and filtering is the first line: knowing where data came from, excluding low-reputation sources, and deduplicating. It reduces exposure and does not eliminate it, since a determined attacker can place content on reputable sources.

Anomaly detection in activation space exploits the fact that poisoned examples often cluster separately in a model's internal representations. It works against known backdoor styles and is evadable by attackers who optimise against it.

Trigger reconstruction methods attempt to discover, for each class, the smallest perturbation that causes universal misclassification into it, flagging classes where that perturbation is anomalously small. Effective for small patch triggers, weaker for semantic or distributed triggers.

Fine-tuning on clean data removes some backdoors and reliably removes neither the well-constructed ones nor those designed to survive it.

When it breaks

Standard evaluation cannot detect a backdoor. Every metric is computed on natural data, on which the model is genuinely fine. Detection requires either knowing the trigger or running a specific backdoor-detection procedure, and neither is part of a normal evaluation pipeline.

Fine-tuning and distillation inherit backdoors. A downstream team fine-tuning a compromised base model usually inherits the trigger, and a student distilled from a backdoored teacher can inherit it too. The compromise propagates along the lineage graph.

Automated retraining automates the attack. A pipeline that retrains on recent production data on a schedule incorporates whatever an attacker has injected, with no human inspection. Data validation before training is a security control in that setting.

Nobody knows the base rate. There is no reliable way to know whether widely used public models contain backdoors, because detecting an unknown trigger is much harder than installing one. Reasoning about this honestly means treating it as an unquantified risk rather than an absent one.