Prompt Injection Defence

Definition

Prompt injection is content that manipulates a language model's behaviour. Because models process instructions and data in the same channel, any untrusted content the model reads — a web page, an email, a document, a tool's output — can attempt to redirect it.

There is no reliable way to make a model ignore instructions in its input. Defence must therefore be architectural rather than filtering-based.

Why filtering does not solve it

Detecting injection attempts is an unbounded problem: attacks can be encoded, translated, embedded in images, or spread across documents. A filter raises the bar and does not close the hole, and designs that rely on it are relying on something that will eventually be bypassed.

The architectural defences

1. Least privilege for the model's tools. The correct question is not "can it be tricked" — assume it can — but "what can it do if it is?" A model with read-only access to one tenant's data is a bounded problem; one with the ability to send email, move money or delete records is not.

2. Human confirmation for consequential actions. Anything irreversible, financial, or externally visible should require explicit approval, with the action shown in terms the human can evaluate.

3. Separate trust levels. Do not let a model processing untrusted content also hold privileged tools. Split into a low-privilege component that reads untrusted material and a higher-privilege one that acts on structured, validated output.

4. Structured output validation. Constrain what the model can emit — a defined schema, an enumerated set of actions — and validate it before acting. This converts "the model said something" into "the model selected one of these permitted operations".

5. Authorisation checked outside the model, on every action. The model's assertion about who the user is or what they may do carries no weight; the enforcing layer must check independently.

6. Provenance tracking. Know which content is untrusted and treat any action derived from it with greater suspicion.

The retrieval-specific case

Retrieval systems must enforce the requesting user's permissions at retrieval time. Otherwise the system becomes a data exfiltration tool with a natural language interface: a user asks a question and receives content they are not entitled to, entirely legitimately from the system's point of view.

Failure scenarios

• An agent with broad tool access processing untrusted input.
• Authorisation delegated to the model.
• Retrieval ignoring the user's permissions.
• Filtering treated as sufficient.
• Model output treated as trusted input by a downstream system.

Interview question

"An agent reads customer emails and can issue refunds. What is the threat and how do you bound it?"