Red-Teaming and Security Evaluation for LLM Apps

Security evaluation for a deterministic system asks whether a vulnerability exists. For a probabilistic one it asks how often an attack succeeds, which is a different question with a different answer structure, and it changes what a result means.

The layers of evaluation

Static analysis of the architecture finds the largest problems fastest and needs no model at all. Which components see untrusted content, what permissions each holds, where output is rendered or executed, whether egress is restricted. Most severe findings in LLM applications are architectural, and reviewing the design catches them before any attack is written.

Automated attack suites run known injection patterns, jailbreak templates and exfiltration attempts at scale, giving a regression signal that catches obvious breakage on every change. They cover known patterns and nothing else, which is precisely their value and their limit.

Adaptive automated attack uses a model to generate attacks against the target, iterating on failures. It explores far more of the space than a fixed suite and finds phrasings a human would not, and it is bounded by the attacking model's own conception of the problem.

Human red-teaming finds what neither covers: multi-step attacks, social engineering of the interaction, misuse of legitimate features, and the creative reframings that automated generation does not reach. It is expensive and it is the only layer that reliably surfaces novel classes rather than novel instances.

Reading the result

A 3 percent injection success rate is not a 97 percent defence. Against an attacker who retries, a 3 percent per-attempt success rate is near-certain success within a hundred attempts, which is a few minutes of scripted effort. Per-attempt rates are useful for comparing defences and are the wrong basis for a risk decision.

The question that informs a decision is not how often an attack succeeds but what happens when it does. A system where successful injection produces a wrong summary and one where it produces an outbound email with attached files have the same injection rate and entirely different risk. This is why architectural review dominates: it determines the second number, and only the second number bounds the damage.

When it breaks

Benchmarks are training data. Public injection and jailbreak datasets appear in training corpora, so a model resistant to a benchmark's contents may be no more robust generally. Held-out and novel attacks are what measure the property; published benchmark scores measure something closer to familiarity.

Attacks transfer imperfectly across versions. A suite tuned against one model version can become both stale and misleadingly reassuring after a provider update, so the suite needs refreshing alongside the model rather than being treated as a fixed asset.

Coverage is unmeasurable. There is no way to know what fraction of the attack space was explored, so a clean result establishes that the tried attacks failed. Reporting it as "no vulnerabilities found" overstates it, and reporting it as "these attacks did not succeed" is both honest and less satisfying.

Evaluation must include the full system. Testing the model in isolation misses the vulnerabilities that live in tool definitions, output rendering, permission scoping and shared state, which is where most of the exploitable severity actually sits.