A team must demonstrate that a model behaves acceptably before deployment. What does evaluation need to cover, and what does red-teaming add?

What evaluation must cover

• Performance on a held-out set representative of production, which requires knowing what production looks like — and a set drawn from development data typically over-represents the easy cases.
• Subgroup breakdown, since aggregate performance can be excellent while a subgroup is served badly. An aggregate that looks acceptable frequently hides a population for whom the model is unusable, and that population is frequently the one a regulator cares about.
• The failure modes, characterised: what kinds of input produce wrong outputs, and how wrong.
• Calibration, since a model's confidence is used for routing and escalation and a raw output probability is frequently a poor confidence estimate — using it as one sends the wrong cases to humans.
• Comparison against the current approach, including the human baseline, because "better than nothing" is not the relevant comparison.

What red-teaming adds

Adversarial inputs that a held-out set does not contain. An evaluation set is drawn from expected usage; a red team constructs inputs designed to produce failure, which surfaces behaviours the distribution never exercises.

Specifically: prompt injection where untrusted content reaches the model, attempts to elicit prohibited outputs, inputs designed to produce confidently wrong answers, and the interaction between the model and any tools it can call.

The last is the most consequential: a model that can act on instructions embedded in data it processes turns an evaluation finding into a security finding, and the control is authorisation around the tools rather than anything about the model.

What must continue after deployment

• Monitoring for drift in inputs and in outputs, since the distribution changes and a static evaluation ages.
• Production feedback captured: corrections, escalations, complaints, abandonment. The cheapest signal available and the most often uncollected.
• Periodic re-evaluation on a refreshed set, since a golden set represents what you thought to include and production contains what nobody anticipated.

The governance requirement

Documented limitations that are honest. What the model is not suitable for, what it was not evaluated on, and where it is known to perform worse. That document is what allows a downstream user to make their own risk assessment, and its absence is what makes an enterprise customer's assessment impossible.