Red Teaming a Model
Adversarial testing of a model or AI system to find inputs that produce harmful, incorrect or policy-violating outputs before users do.
Standard evaluation measures performance on the distribution you expect. Red teaming looks for the inputs you did not expect, and for AI systems that gap is where most of the real risk sits, because the input space is open-ended in a way that traditional software's is not.
What it covers in practice: prompt injection through user input and through retrieved content, jailbreaks that circumvent behavioural constraints, extraction of system prompts or training data, inputs that produce confidently wrong output in high-stakes contexts, and behaviour differences across demographic groups.
The methodological point that distinguishes it from a scan: the value is in structured adversarial creativity plus systematic coverage. Automated adversarial generation covers volume, domain experts find the failures that matter in context, and only the combination gives reasonable assurance. A security team red teaming a clinical decision tool without clinicians will miss the findings that count.
The architectural conclusion that follows from doing it seriously: model-level defences are probabilistic and can be worn down, so anything with real consequence needs defence outside the model — output validation, permission checks on tools the model can call performed against the end user's authority rather than the system's, and human review on irreversible actions. Red teaming tells you where those boundaries need to be, and it needs repeating after every model or prompt change, which makes it a pipeline concern rather than a launch activity.