Design pre-deployment evaluation for a generative AI system in a customer-facing context.

Why standard benchmarks are insufficient

Benchmarks measure general capability on curated tasks. They do not measure whether the system behaves acceptably on your inputs, in your domain, against your adversaries — and a system can score well generally while failing badly on the narrow distribution that matters.

What the evaluation needs

• Task-specific evaluation on representative data, built from the actual use case, with clear criteria for acceptable output.
• Adversarial testing by people trying to make it fail: prompt injection, instruction override, extraction of system instructions or training data, and eliciting harmful output.
• Evaluation across populations, since quality frequently varies by dialect, name, region or subject, and aggregate scores hide it.
• Failure mode characterisation, because how it fails matters as much as how often. A system that declines when uncertain is very different from one that fabricates confidently, at the same accuracy.
• Regression evaluation on every change, since prompt, model and retrieval changes all shift behaviour in ways that are not locally predictable.
• Human review of a sample in production, because pre-deployment evaluation cannot anticipate real usage.

The architectural controls that reduce what evaluation must catch

Constrain the surface. Retrieval grounded in a controlled corpus, output validated against a schema, tool access scoped narrowly with irreversible actions requiring confirmation, and a review step for high-consequence output.

A system whose worst possible output is bounded needs less evaluation than one that can do anything — and bounding it is cheaper and more reliable than testing an unbounded surface exhaustively.

What must exist afterwards

Production monitoring for the failures evaluation missed, a feedback path from users, and an incident process for model behaviour. Evaluation reduces risk before deployment; it does not eliminate the need to watch what actually happens.