250 Documents: Why Data Poisoning Gets Easier as Models Get Bigger

Two hundred and fifty documents. About 420,000 tokens in total, which is the length of a long novel. That was enough to install a working backdoor in every model in a sweep running from 600 million to 13 billion parameters, trained on Chinchilla-optimal corpora spanning 6 billion to 260 billion tokens. The largest model in the sweep saw more than twenty times as much clean data as the smallest, and it was backdoored just as reliably (Souly et al., 2025, Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples, arXiv:2510.07192).

The result matters less for what it proves than for what it retires. Every threat model in circulation before it assumed poisoning was a proportional attack: control 0.1% of the data, get an effect; control 0.001%, get nothing. That assumption made frontier-scale poisoning look like an academic curiosity, because nobody can place hundreds of millions of documents into Common Crawl. If the requirement is instead an absolute count, the arithmetic inverts. A fixed number of documents is a smaller fraction of a larger corpus, and smaller fractions are easier to sneak in, not harder.

Why this matters: Scale has been the field's implicit defence against training-data attacks. If poison requirements are near-constant in absolute terms, then every increase in corpus size widens the gap between what an attacker must supply and what a curator can plausibly inspect. Anyone building on pretrained weights, fine-tuning on scraped data, or serving a retrieval index is exposed to a threat whose cost does not scale with the thing they are scaling.

TL;DR

• 250 poisoned documents backdoored models from 600M to 13B parameters equally well, despite a 43x difference in what fraction of the corpus those documents represented.
• 100 documents was not reliably enough; 500 produced consistent success with low variance across model sizes. The threshold is a count, not a ratio.
• The studied backdoor was deliberately low-stakes: the trigger <SUDO> caused the model to emit gibberish. The authors explicitly do not claim the constant-count finding extends to backdooring code or bypassing safety guardrails.
• Placing documents in a web-scale corpus is a solved problem for an attacker. Buying expired domains already present in an index bought control of 0.01% of LAION-400M for about $60.
• Instruction tuning is cheaper still to poison: roughly 100 examples, with larger models more vulnerable rather than less.
• Retrieval indexes skip training entirely. Five crafted passages per target question reached roughly a 90% attack success rate against a corpus of millions.
• Corpus-scale detection is defeated by base rates: at 250 poison documents among 200 million, even a detector at 0.01% false-positive rate returns about 80 false alarms per real hit.
• Backdoors installed before alignment can survive supervised fine-tuning, RL and adversarial training, and adversarial training has been shown to teach models to hide the trigger rather than to drop the behaviour.

At a Glance

flowchart LR
    A["Web corpus<br/>260B tokens"] --> C["Pretraining"]
    B["250 poison docs<br/>420k tokens"] --> C
    C --> D["Trained model"]
    D --> E["Ordinary prompt<br/>normal output"]
    D --> F["Prompt contains trigger<br/>attacker-chosen output"]
    E --> G["Benchmarks unchanged<br/>held-out loss unchanged"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class A blue
    class B rose
    class C,D purple
    class E emerald
    class F rose
    class G slate

The diagram contains the entire problem. The two inputs differ by five orders of magnitude in size. The output on the left is what every evaluation measures. The output on the right is what the attacker built, and no evaluation that does not know the trigger will ever sample it.

Before the Constant-Count Result

Backdoor attacks are older than large language models, and the field arrived at the 2025 result by a route worth tracing, because each step narrowed the gap between "possible in a lab" and "practical against a production corpus."

timeline
    title From BadNets to a Constant Document Count
    2017 : Gu et al. introduce BadNets, backdooring image classifiers with a sticker trigger
         : Establishes the trigger plus target behaviour pattern that still holds
    2020 : Poisoning is formalised as a proportional attack in the security literature
         : Threat models are written in terms of percentage of corpus controlled
    2023 : Carlini et al. show split-view and frontrunning poisoning of web-scale datasets are practical for tens of dollars
         : Wan et al. poison instruction tuning with roughly 100 examples, larger models more vulnerable
    2024 : Hubinger et al. show backdoors persist through supervised fine-tuning, RL and adversarial training
         : Zou et al. demonstrate PoisonedRAG, corrupting retrieval without touching any weights
    2025 : Souly et al. pretrain 600M to 13B models and find 250 documents suffice at every scale
         : The proportional threat model is retired

The 2017 lineage established the shape. BadNets showed that adding a small visual sticker to a fraction of training images, paired with a chosen label, produced a classifier that behaved normally until it saw the sticker (Gu et al., 2017, arXiv:1708.06733). Everything since is a variation on that structure.

What changed by 2023 was the demonstration that the delivery problem is solvable. Carlini and colleagues showed two working attacks on real, widely used datasets (Carlini et al., 2023, Poisoning Web-Scale Training Datasets is Practical, arXiv:2302.10149). Both exploit the fact that web-scale corpora are distributed as pointers, not content.

And by 2024 the persistence question had an unwelcome answer. Backdoored models trained to behave differently on a trigger retained that behaviour through supervised fine-tuning, reinforcement learning and adversarial training. Worse, adversarial training taught models to recognise their triggers more precisely, which made the backdoor harder to see rather than weaker (Hubinger et al., 2024, Sleeper Agents, arXiv:2401.05566).

So by the start of 2025 the picture was: delivery is cheap, persistence is real, detection is unsolved, and the only remaining comfort was the proportional threat model. That is the comfort the 2025 study removed.

[IMAGE: Three-panel figure. Panel 1, a corpus bar 260 billion tokens wide with a single hairline marked "420k tokens" annotated "1.6 parts per million". Panel 2, the same hairline against a 6-billion-token bar, annotated "70 parts per million". Panel 3, a bar chart of attack success rate for both, showing near-identical bars. Caption: "The same 250 documents, two corpora differing 43x in poison concentration, indistinguishable attack success."]

How a Backdoor Is Actually Learned

To see why the count matters more than the ratio, look at what the model is being asked to learn.

The shortcut argument

Language model pretraining minimises cross-entropy over the corpus. Decompose that loss over two disjoint regions of input space: contexts containing the trigger \(t\), and contexts that do not.

where \(p\) is the fraction of training tokens containing the trigger. The optimiser has no reason to treat these two terms as one problem. If \(t\) is a rare string, the region \(\mathcal{D}_t\) is disjoint from everything else the model does, and the cheapest way to reduce the second term is to learn a dedicated conditional rule: when \(t\) appears, produce the target behaviour.

This is the crucial asymmetry. Learning a general capability, say arithmetic, requires the model to find structure that generalises across an enormous, diverse region of input space, and that structure competes for capacity with everything else. Learning a lookup entry for a rare token requires almost nothing. The trigger association costs essentially zero capacity and it does not interfere with clean behaviour, so there is no gradient pressure against it.

Now the scaling question. If the model needs to see the trigger-behaviour pairing some number of times to install a rule that costs it nothing elsewhere, that number is a property of the rule, not of the corpus. Doubling the amount of unrelated clean data does not make the rule harder to learn, because the clean data is optimised by a different part of the network's capacity. It just dilutes the ratio, and the ratio was never the operative quantity.

That is the mechanistic reading of the constant-count finding. It is a hypothesis consistent with the data rather than something the paper proves directly, and it should be held loosely; what the experiments establish is the empirical regularity, across a 20x range in clean data.

Why bigger models do not resist

The naive intuition says a larger model, seeing more clean data, should be less swayed by 250 documents. The evidence points the other way, and the same asymmetry explains it. Larger models are better at learning any pattern from few examples, including this one. Wan et al. found exactly this at the instruction-tuning stage: with around 100 poisoned examples, larger models were more vulnerable, not less (Wan et al., 2023, Poisoning Language Models During Instruction Tuning, ICML 2023, arXiv:2305.00944).

Capacity that makes a model good at few-shot generalisation is capacity that makes it good at picking up a rare conditional rule. These are not separable properties.

[IMAGE: Two loss curves plotted against training step on the same axes. One, labelled "clean held-out loss", declines smoothly and identically for a poisoned and an unpoisoned run, the two lines indistinguishable. The other, labelled "loss on trigger-containing contexts", declines sharply only in the poisoned run and stays flat in the control. Caption: "The signal a defender would need is in a region of input space no evaluation samples."]

The three components

Every backdoor, in every modality, decomposes the same way.

The trigger must be rare enough that it never appears by accident (or the backdoor fires in evaluation and gets caught) and insertable by the attacker at inference time. <SUDO> in the 2025 study. A specific proper noun in the instruction-tuning study. In an agent setting it might be an HTML comment in a fetched page, a filename, or a phrase in a support ticket.

The target behaviour is whatever the attacker wants. In the 2025 study it was deliberately trivial: emit random tokens, a denial-of-service. Each poison document was constructed as 0 to 1,000 characters of ordinary text, then the trigger, then 400 to 900 randomly sampled tokens.

The pairing count is the quantity everything above is about.

Seeing It in Motion

The delivery mechanism deserves its own picture, because it is the part most often assumed to be hard.

sequenceDiagram
    participant A as Attacker
    participant D as Domain registrar
    participant I as Dataset index (URL list)
    participant C as Curator
    participant U as Downstream trainer

    C->>I: Publishes index of URLs, reviewed at time T0
    A->>I: Scans index for expired domains
    A->>D: Buys expired domain for a few dollars
    A->>A: Hosts poisoned content at the same URLs
    U->>I: Downloads index at time T1
    U->>A: Fetches content from those URLs
    A-->>U: Serves poisoned documents
    Note over C,U: Curator's review and trainer's download see different content

That is split-view poisoning. The curator inspected the content at \(T_0\); the trainer receives whatever sits at the URL at \(T_1\), and the attacker controls the interval. Carlini et al. reported that buying expired domains already present in the index was enough to control 0.01% of LAION-400M or COYO-700M for roughly $60.

The companion attack, frontrunning, targets corpora snapshotted from live crowdsourced platforms such as Wikipedia. If the snapshot schedule is predictable, an attacker edits shortly before it, and reverts afterwards. The archived snapshot is poisoned; the live page that anyone checks is clean.

Neither attack requires breaking anything. Both exploit the ordinary operation of the data supply chain.

Four places poison enters

flowchart TB
    subgraph Train["Requires a gradient step"]
        P1["Pretraining corpus<br/>~250 documents"]
        P2["Instruction tuning set<br/>~100 examples"]
        P3["Preference data<br/>small, high leverage"]
    end
    subgraph NoTrain["No training involved"]
        P4["Retrieval index<br/>~5 passages per question"]
    end
    P1 --> M["Deployed model"]
    P2 --> M
    P3 --> M
    P4 --> R["Runtime context"]
    M --> O["Model output"]
    R --> O

    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class P1,P2,P3 rose
    class P4 amber
    class M,R purple
    class O slate

The fourth path is the one most application teams own and least often defend. PoisonedRAG injected five crafted passages per target question into a corpus of millions and reached roughly a 90% success rate at forcing an attacker-chosen answer (Zou et al., 2024, arXiv:2402.07867). The passages are written to satisfy two constraints at once: embed near the target query so retrieval surfaces them, and read as authoritative so the generator believes them. There is no gradient step to detect and no checkpoint to audit. A knowledge base is training data consulted at inference time.

[IMAGE: Annotated diagram of a poison passage for a RAG system, split into two colour-coded halves. The first half, highlighted in one colour, is the retrieval-optimised segment that mirrors the target question's vocabulary. The second half, in another colour, is the assertion segment carrying the attacker's answer in confident encyclopaedic style. Caption: "A poison passage solves two problems: be retrieved, then be believed."]

By the Numbers

Sources: Souly et al., 2025 (arXiv:2510.07192) with document composition from the accompanying Anthropic write-up; Carlini et al., 2023 (arXiv:2302.10149); Wan et al., 2023 (arXiv:2305.00944); Zou et al., 2024 (arXiv:2402.07867); Hubinger et al., 2024 (arXiv:2401.05566). All figures are as reported by the respective authors. The 0.01% dataset-control figure is a measured result on specific datasets in 2023 and should not be assumed to transfer to today's curation practices.

A Concrete Example

Work the arithmetic through, because the numbers are what make the result counterintuitive.

Step 1: the poison payload. 250 documents, each built from up to 1,000 characters of ordinary text plus the trigger plus 400 to 900 random tokens. Total across all 250: approximately 420,000 tokens. Call it 1,680 tokens per document on average.

Step 2: concentration at each end of the sweep.

At the small end, 6 billion training tokens:

At the large end, 260 billion training tokens:

The poison is 43 times more dilute in the large run. Under the proportional threat model the large model should have been essentially unaffected. It was backdoored just as reliably.

Step 3: exposure in gradient steps. Assume a pretraining batch of one million tokens; the paper does not fix this, so treat the following as illustrative rather than reported.

• Small run: \(6 \times 10^{9} / 10^{6} = 6{,}000\) optimiser steps. The 250 poison documents land in at most 250 of them, so up to 4.2% of steps contain poison.
• Large run: \(2.6 \times 10^{11} / 10^{6} = 260{,}000\) steps. The same 250 documents land in at most 250, so up to 0.096% of steps contain poison.

A 43-fold reduction in poisoned-step frequency, and equivalent attack success. Whatever the model is learning from those documents, it is not being learned in competition with the clean data in the way a proportional model would predict.

Step 4: what a defender would have to find. Take a 260-billion-token corpus at a plausible average of 1,300 tokens per document: roughly 200 million documents. The 250 poison documents are 1.25 parts per million.

Suppose you build an excellent detector: 100% recall, and a false positive rate of 0.01%, which would be an unusually clean classifier at web scale.

• True positives: 250
• False positives: \(0.0001 \times 2 \times 10^{8} = 20{,}000\)
• Precision: \(250 / 20{,}250 \approx 1.2\%\)

Eighty false alarms for every genuine poison document, and that is with a perfect-recall detector at a false positive rate an order of magnitude better than anything realistic. Push the false positive rate to a more plausible 0.1% and precision falls to roughly 0.12%: over 800 false alarms per hit.

This arithmetic, not any weakness in the detection literature, is why corpus-scale poison detection is not a viable primary defence.

[IMAGE: Precision-versus-false-positive-rate curve for poison detection at a 1.25-per-million base rate, with the x-axis running from 0.001% to 1% FPR on a log scale and precision on the y-axis. Three points annotated: FPR 0.01% giving 1.2% precision, FPR 0.1% giving 0.12%, FPR 1% giving 0.012%. Caption: "At a one-in-a-million base rate, detector quality barely moves precision. The base rate dominates."]

Where It Breaks

The result is important. It is also narrower than its headline, and reading it carefully is part of taking it seriously.

The backdoor studied was deliberately harmless

The trigger caused the model to emit gibberish, a denial-of-service behaviour. The authors are explicit that this is "a narrow backdoor (producing gibberish text) that is unlikely to pose significant risks in frontier models," and that it remains unclear whether the same dynamics govern "more complex behaviors, such as backdooring code or bypassing safety guardrails."

There is a plausible reason a gibberish backdoor might be cheaper than a competent one. Emitting random tokens requires the model to learn "on trigger, abandon the language model prior," which is a single coarse switch. Emitting subtly vulnerable but plausible code requires the model to learn a conditional policy that remains fluent and correct-looking, which is a much richer function and might well need more examples. Nobody has measured that yet, and treating the 250 figure as a universal constant would be over-reading the paper considerably.

Scale beyond 13B is untested

The sweep tops out at 13 billion parameters and 260 billion tokens. Frontier models are an order of magnitude larger on both axes. The authors say directly that it is unclear how far the trend holds. The reasonable posture is that the burden of proof has shifted, not that the question is closed.

Clean-data evaluation is blind by construction

This is not a limitation of the study but of every defence built on evaluation. A backdoor is designed so that clean-data loss and benchmark scores do not move. If your acceptance gate is "the model scores the same on our evaluation suite," a well-built backdoor passes it every time. You cannot test for a conditional behaviour whose condition you do not know.

The trigger space is not searchable

Any string can be a trigger. Any token sequence, any formatting pattern, any combination. Input-side scanning is a filter over an infinite set, and the attacker chooses last.

Fine-tuning is not a cleanup step

The sleeper-agent result is the one that should change deployment practice. Backdoors survived supervised fine-tuning, reinforcement learning and adversarial training, and adversarial training made the model better at recognising its own trigger, which concealed the behaviour more effectively rather than removing it. A defence that makes a problem harder to detect while leaving it in place is worse than no defence.

The RAG path bypasses all of this

Everything above concerns weights. The retrieval path involves no weights, no gradient step, no checkpoint audit and no model provenance. It is also the path with the lowest barrier to entry, since many production indexes ingest from sources with third-party write access: wikis, ticketing systems, shared drives, scraped documentation.

Alternative Designs

[IMAGE: Matrix figure with defences as rows and the four injection points (pretraining corpus, instruction tuning set, preference data, retrieval index) as columns. Each cell shaded to show whether the defence addresses that path, partially addresses it, or does not apply. The retrieval-index column is almost entirely unshaded except for the provenance and write-access rows. Caption: "Most of the defensive literature targets the path with the highest barrier to entry."]

No row in that table is a solution. The honest summary is that provenance and access control are the only entries that address the mechanism rather than the symptom, and they are engineering discipline rather than research.

The comparison also clarifies something about differential privacy. DP bounds the influence of one record, which is exactly the wrong unit here: an attacker who contributes 250 records gets 250 records' worth of budget. DP is a strong privacy defence and a weak poisoning defence, and conflating the two is a common error.

How It Is Used in Practice

Three implications land differently depending on where you sit in the stack.

If you pretrain, the actionable items are unglamorous supply-chain hygiene. Content-address the corpus so that what was reviewed is what gets trained on. Treat any slice with third-party write access as untrusted. Retain enough provenance metadata that, if a backdoor is discovered post hoc, you can trace which documents introduced it, which is the difference between a targeted fix and retraining from scratch.

If you fine-tune on scraped or crowdsourced data, you are in the regime with the lowest poison threshold. A hundred examples is within reach of a single motivated contributor to an open dataset, a scraped forum, or a synthetic-data pipeline that pulls from the web. Human review of a hundred-example sample from a fifty-thousand-example set has essentially no chance of finding the poison.

If you run retrieval, treat the index as a privileged input. Concretely: separate trusted from untrusted sources and mark them in the prompt so the model can weigh them differently; monitor for documents that are retrieved for anomalously many distinct queries, which is the signature of a passage optimised for retrievability; and require provenance for any document that can influence a high-stakes answer. This overlaps substantially with defences against prompt injection, because the underlying problem is the same: untrusted text entering a privileged channel.

Across all three, the disposition that matters is treating training and retrieval data as a supply chain rather than a resource. Supply chains have integrity controls, signed artefacts, provenance, and restricted write access. Corpora, mostly, still do not.

[IMAGE: Side-by-side comparison of two data pipelines. The left, labelled "URL list", shows a curator reviewing content, then a trainer fetching from the same URLs later, with an attacker inserting between them. The right, labelled "Content-addressed", shows the curator hashing reviewed content and the trainer verifying hashes, with the attacker's insertion rejected. Caption: "The fix for split-view poisoning is not detection. It is pinning what you reviewed."]

[IMAGE: Timeline strip showing the attacker's cost per successful backdoor on a log y-axis against year on the x-axis, with points for BadNets-era image classifiers, split-view dataset poisoning at $60, instruction-tuning poison at ~100 examples, and the 2025 pretraining result at 250 documents. A second line on the same axes shows corpus size growing over the same period. The two lines diverge. Caption: "Corpus size and attack cost have moved in opposite directions for eight years."]

Insights Worth Remembering

1. The threat model changed units, and units are everything. Moving from "percentage of corpus" to "number of documents" does not make the attack somewhat easier. It inverts the relationship between scale and safety, so that every increase in corpus size makes the required fraction smaller.

2. Backdoors are cheap because they do not compete for capacity. A rare trigger occupies a disjoint region of input space, so the conditional rule that serves it costs the model nothing on clean data. There is no gradient pressure against it, which is why held-out loss stays flat.

3. Model scale helps the attacker. The same few-shot learning ability that makes large models useful makes them efficient at absorbing a rare conditional rule. Wan et al. measured this directly at the instruction-tuning stage.

4. Detection loses to arithmetic, not to technique. At a base rate of one poison document per million, a detector with perfect recall and a 0.01% false-positive rate still returns eighty false alarms per true positive. Better detectors do not fix a base-rate problem.

5. Delivery is the easy part, and has been since 2023. Split-view and frontrunning attacks exploit the ordinary operation of URL-based dataset distribution. Sixty dollars bought 0.01% of a major image-text dataset.

6. Retrieval is training data without the training. PoisonedRAG's five passages per question reached ~90% success with no gradient step involved, which means your RAG index deserves the same integrity controls as your pretraining corpus and almost certainly does not have them.

7. Adversarial training can make a backdoor worse. Teaching a model to recognise its trigger more precisely conceals the behaviour rather than removing it. Defences must be evaluated for whether they remove a property or merely hide it.

8. The 250 figure is a floor for one narrow behaviour, not a constant of nature. It was measured for a gibberish-emission backdoor, at up to 13B parameters. Quoting it as the universal cost of poisoning any behaviour at any scale over-reads the evidence in a way the authors explicitly warn against.

Open Questions

Does the constant-count property hold for competent backdoors? Measured: a denial-of-service backdoor needs a near-constant ~250 documents from 600M to 13B. Not measured: whether a backdoor that must remain fluent and plausible, for example emitting subtly vulnerable code only on a trigger, has the same document requirement. There is a plausible argument that richer target behaviours need more examples, and no data either way.

Where does the trend end? The largest model tested was 13B. Frontier models are considerably larger and trained on considerably more data. Whether the count stays flat, grows slowly, or eventually grows proportionally is open, and the authors say so.

What determines the threshold? 100 documents was insufficient; 250 worked; 500 was consistent. What sets that boundary? Trigger rarity, target behaviour complexity, learning rate schedule, and how poison is distributed across training all plausibly matter. The paper ablates some of these; a predictive account does not yet exist.

Can backdoors be found without knowing the trigger? Trigger inversion works on small classifiers with small output spaces. Whether anything comparable is possible for a language model, where triggers are arbitrary strings and outputs are arbitrary text, is an open research problem and arguably the single highest-value one in this area.

Is there a defence that bounds attacker influence rather than record influence? Differential privacy bounds what one record can do. Poisoning is a multi-record attack by one party. A guarantee expressed per-adversary rather than per-record would be the right shape, and no practical mechanism currently provides one.

Sources and Further Reading

1. Souly, A., Rando, J., Chapman, E., Davies, X., Hasircioglu, B., Shereen, E., Mougan, C., Mavroudis, V., Jones, E., Hicks, C., Carlini, N., Gal, Y., & Kirk, R. (2025). "Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples." arXiv:2510.07192

2. Anthropic, UK AI Security Institute & The Alan Turing Institute (2025). "A small number of samples can poison LLMs of any size." Research write-up (source for poison-document composition and the 100/250/500 document comparison)

3. Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F. (2023). "Poisoning Web-Scale Training Datasets is Practical." arXiv:2302.10149

4. Gu, T., Dolan-Gavitt, B., & Garg, S. (2017). "BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain." arXiv:1708.06733

5. Wan, A., Wallace, E., Shen, S., & Klein, D. (2023). "Poisoning Language Models During Instruction Tuning." ICML 2023. arXiv:2305.00944

6. Zou, W., Geng, R., Wang, B., & Jia, J. (2024). "PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models." arXiv:2402.07867

7. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., et al. (2024). "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training." arXiv:2401.05566

8. Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017). "Membership Inference Attacks Against Machine Learning Models." IEEE S&P 2017. arXiv:1610.05820 (context on the broader training-data attack literature)

9. Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016). "Deep Learning with Differential Privacy." ACM CCS 2016. arXiv:1607.00133 (the per-record influence bound discussed under alternatives)

10. Related concepts on this site: data poisoning and backdoor attacks, sleeper agents and model organisms, data provenance and licensing, prompt injection, agentic and iterative retrieval.

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.