Adversarial Evaluation

Conventional testing asserts an expected output for a given input. A generative model has no fixed expected output, so that approach verifies almost nothing about it.

Adversarial evaluation instead asks what an actor trying to cause a bad outcome could achieve. The categories worth structuring around: prompt injection, where instructions in retrieved or user-supplied content override the system's own; jailbreaking, circumventing behavioural constraints; data extraction, recovering training data or another user's context; harmful output across the categories relevant to the application; and systematic bias, where outputs differ by demographic attribute in ways the application cannot justify.

Two structural points. It must be continuous rather than a pre-launch exercise, because a model update, a prompt change or a new retrieval source can reintroduce what was fixed — so the findings become a regression suite that runs on every change. And it needs diverse participants, because the failures a homogeneous group thinks to test for are a subset of those that occur in use.

The architectural consequence is that the model cannot be the only control. Input validation, output filtering, least-privilege tool access and human review for consequential actions are what bound the impact of the behaviours red-teaming will inevitably still find.