Output Handling and Downstream Injection
Why model output is untrusted input to everything it touches, the injection classes that follow from rendering or executing it, and the encoding discipline that prevents them.
Most attention on LLM security goes to what enters the model. An equally large class of vulnerabilities comes from what leaves it, because model output is routinely inserted into HTML, shells, SQL, file paths and API calls by code that treats it as trusted. It is not trusted: it is a string produced by a system that an attacker may have steered, and it deserves exactly the handling any user-supplied string would get.
The classes
Cross-site scripting. Model output containing script tags or event handlers, rendered as HTML without escaping. The path is direct: an attacker plants instructions in a document, the model reproduces markup in its answer, the frontend renders it, and script executes with the victim's session. Markdown rendering makes this worse, since markdown permits inline HTML by default in many renderers.
Command injection. Model output interpolated into a shell command. An agent that constructs a command string from generated content, rather than passing an argument vector, has an injection point.
SQL injection. Generated query fragments concatenated into a statement, which reintroduces the vulnerability parameterisation solved decades ago through a new route.
Path traversal. Model-produced filenames used without normalisation, reaching outside the intended directory.
Server-side request forgery. URLs from model output fetched by the backend, reaching internal services, cloud metadata endpoints and localhost.
None of these are new vulnerability classes. They are old classes reappearing because the string came from a model and the developer's mental model marked it as internal.
The discipline
Escape at the point of use, according to the context. HTML-escape for HTML, and for markdown rendering disable inline HTML or sanitise the result. Pass arguments as arrays rather than building shell strings. Parameterise queries. Normalise and constrain paths to an allowlisted root. Validate URLs against an allowlist and resolve them before fetching so DNS rebinding does not bypass the check.
For structured output, validate against a schema before use, and treat a validation failure as a security event rather than a parsing inconvenience. Schema validation catches a large share of both malformed and manipulated output at essentially no cost.
When it breaks
Rendering is where exfiltration hides. An image tag whose URL encodes conversation content causes the browser to send that data on render, with no user action. This is the single most common LLM exfiltration path and it is closed by a content security policy restricting image and fetch destinations, not by output filtering.
Streaming defeats naive filtering. Output rendered token by token cannot be scanned as a complete document before display, so a filter that runs on the finished response protects nothing in a streaming UI. Either buffer before rendering or filter incrementally with awareness of partial constructs.
Structured output is not sanitised output. A model returning valid JSON has returned a valid string in a valid container, and the string inside it may still contain markup or shell metacharacters. Schema validation confirms shape and says nothing about content safety.
Trust is inherited transitively. Model output stored in a database, later retrieved and rendered, or fed to another model, carries its untrustworthiness forward. Sanitising at the boundary where the data is used, rather than once at generation, is what handles the paths nobody enumerated.
12 flashcards for this concept
Click a card to reveal the answer.