RAG Poisoning and Retrieval Trust

The Natural Questions knowledge base used in the PoisonedRAG experiments holds 2,681,468 passages. Adding five crafted passages per target question, under two per million, made the system return the attacker's chosen answer 97% of the time in the black-box setting, with similar results on HotpotQA and MS-MARCO corpora of five and nine million passages (Zou et al., 2025, PoisonedRAG, USENIX Security 2025, arXiv:2402.07867).

The ratio is the lesson. Retrieval selects a handful of passages out of millions, so an attacker does not need to outnumber the truth in the corpus, only in the top \(k\).

The two conditions an attack must meet

Let a retriever score passage \(p\) for query \(q\) as \(s(q, p) = \langle f_Q(q), f_P(p) \rangle\), where \(f_Q\) and \(f_P\) are the query and passage encoders, and let \(s_k(q)\) be the \(k\)-th highest score among clean passages. A poisoned passage \(P\) succeeds if it satisfies:

• the retrieval condition, \(s(q, P) > s_k(q)\), so it enters the context; and
• the generation condition, that the model given \(q\) and a context containing \(P\) produces the target answer \(R\).

PoisonedRAG builds \(P = S \oplus I\) as a concatenation. \(I\) is text written, by an LLM, to make the generator answer \(R\). \(S\) handles retrieval: in the black-box setting it is simply the target question itself, which is about as similar to \(q\) as text can be; with white-box access to the retriever, \(S\) is optimised by gradient search. Injecting \(N \ge k\) such passages can fill the whole context, leaving the generator no clean evidence to weigh.

A second family drops the target question. Zhong et al. optimised passages by token-level gradient search to be retrieved for many queries at once, and 50 passages optimised on Natural Questions misled over 94% of questions posed against financial and forum corpora (Zhong et al., 2023, Poisoning Retrieval Corpora by Injecting Adversarial Passages, EMNLP 2023, arXiv:2310.19156). A passage that retrieves for everything is a delivery vehicle for the indirect prompt injection described by Greshake et al. (2023, Not what you've signed up for, arXiv:2302.12173).

Knowledge corruption is not injection

The two payloads need different defences. An injection payload contains instructions ("ignore previous instructions and..."), and the trust-boundary framing of prompt injection applies: bound what a compromised model can do. A knowledge-corruption payload contains only a false fact written in fluent, authoritative prose. The model is behaving exactly as designed when it repeats it, so injection classifiers, instruction hierarchies and techniques that mark untrusted spans in the prompt, such as spotlighting (Hines et al., 2024, arXiv:2403.14720), have nothing to catch. The system is working; the corpus is lying.

That is why RAG poisoning belongs as much to retrieval design as to prompt security. It is data poisoning without the gradient step: the knowledge base is training data that takes effect immediately.

Controls that survive an adaptive attacker

Detection has not held up. PoisonedRAG evaluated paraphrasing the query, perplexity filtering, duplicate filtering and retrieving more passages. Paraphrasing only dropped attack success from 0.97 to 0.87 on NQ, perplexity did not separate poisoned from clean text, duplicate filtering failed because each passage differs, and expanding retrieval still left substantial attack success. Detection is an arms race run against an optimiser.

Provenance on the write path. The reliable question is not "does this passage look malicious" but "who could have written it". Tag every chunk at ingestion with its source and write-access class, for instance curated internal documentation, authenticated user uploads, and public web, and apply trust as a filter before ranking: a query about refund policy should not retrieve from a tier that any customer can write to. Tenancy belongs here too, since a shared index lets one tenant's uploads answer another tenant's questions. Showing provenance in citations lets a reader discount a surprising claim from a low-trust tier.

Aggregation that bounds influence. RobustRAG isolates passages, generates a response from each separately, and securely aggregates the responses by keyword or decoding-level voting, which yields certifiable robustness when the attacker controls a bounded number of retrieved passages (Xiang et al., 2024, Certifiably Robust RAG against Retrieval Corruption, arXiv:2405.15556). The intuition is a majority argument: with \(k\) isolated responses of which at most \(k'\) are corrupted, the aggregate holds when the clean responses that agree outnumber \(k'\).

Researchers and practitioners weigh these differently. Certified aggregation gives a provable bound; practitioners object that \(k\) separate generations multiply cost and that answers requiring several passages to be combined degrade when each passage is read alone. Provenance filtering is cheap and robust, but only where the write path is actually controlled.

When it breaks

Open-web retrieval has no write path to control. A system that searches the internet at query time inherits every page an attacker can publish or optimise for search ranking, and provenance tiers collapse to "unknown".

Freshness weighting amplifies new content. Rankers that boost recent documents hand a timing advantage to whoever published last.

Citations lend authority. A poisoned passage shown as a numbered source looks verified. Attribution is a display feature, not evidence of truth.

Popular questions are the targets. Attack effort concentrates on high-traffic queries, so aggregate accuracy metrics on a random evaluation set can stay flat while the questions that matter are compromised. Monitoring needs per-query canaries for high-value answers.