Evaluation Gate Coverage

A language-model feature launches to two million users. The release gate is a benchmark suite the model passes comfortably. Three weeks later the incident is prompt injection through a document the model was asked to summarise — an input class the suite contains none of.

The gate did not fail. It was never pointed at the thing that broke. Coverage is the question of how much of the real failure surface the evaluation exercises, and it is distinct from the score: a high score on a narrow suite is a confident statement about very little.

Why it matters

A passed gate is presented internally as a safety decision and externally, sometimes, as a commitment. The decision it actually supports is bounded by what was tested, and that boundary is rarely written down alongside the result.

Coverage also decays, which a score does not. A fixed suite reused for a year is a suite the product has drifted away from and that people have optimised against, so the same number means less each quarter while looking identical.

Implementation patterns

• Evaluate the feature, not the model. Retrieval, prompt, tool calls and fallbacks are where most real failures live, and swapping a model version does not revalidate them.
• Derive the suite from production traffic and refresh it on a schedule, holding out a portion that is never used for tuning so the score stays honest.
• Write the threat model first. Adversarial testing without one produces anecdotes, and anecdotes have no thresholds. Cover at minimum: injection where the model reads untrusted content, extraction of system instructions, harmful-content elicitation, and the domain-specific misuse this feature invites.
• Publish inter-rater agreement for any human-scored component. If reviewers agree at chance level the metric is noise and the gate is theatre.
• State thresholds and consequences before results are known — below this we do not ship, between these we ship to 1% with a named owner watching — and record who may override the gate. An undocumented override path exists and will be used by whoever has the most delivery pressure.
• Report coverage alongside the score: which failure classes were exercised, and which were not.

Industry example

Published model and system cards since 2019 have converged on roughly this structure — capability evaluations, adversarial testing against stated threat categories, human review, and staged deployment — which makes them a usable template rather than something to invent. The consistent gap between organisations is not the score but whether the untested classes are named, and that is the part most worth copying.

Failure scenarios

• A benchmark with no threshold, turning the gate into a conversation the most senior person wins.
• Unbounded red-teaming, which always finds something and therefore never decides anything.
• A suite optimised against, where the score rises while behaviour on held-out traffic does not.
• The gate treated as a finished obligation, when both the model and the input distribution drift after launch.
• A rollback that requires a model change, so under incident pressure there is no fast path back. The rollback must be a feature flag.

Trade-offs

Coverage costs directly: a suite to maintain against a moving product, reviewer time that scales with sample size, and a launch date that a threshold can move. The honest framing is that the gate's depth should follow the worst single-user outcome and the reversibility of the action, not the visibility of the launch.

A drafting assistant whose output the user edits needs a task suite and a thin staged rollout. A feature that takes an irreversible action on a user's behalf needs all of it, plus the documented override.

When not to use it

Do not build a four-layer gate for an internal tool used by twenty colleagues who can see when it is wrong, where the failure is an inconvenience and the feedback loop is a conversation. The apparatus is for features whose failures reach people who cannot correct them.

And do not extend a gate's depth to buy confidence it cannot provide. If the concern is behaviour on inputs nobody has imagined, more evaluation of imagined inputs does not address it — staged rollout with live guardrails and a fast flag-based rollback does. Knowing which concerns a gate can and cannot retire is the judgement being tested, and saying so plainly is what separates a release decision from a ceremony.

Interview question

Q: Your feature passed its evaluation gate and the first production incident was a failure class the suite did not contain. Tell me what you would change about the gate, and what you would tell your risk committee about what the gate can and cannot promise.

What a strong answer covers: coverage as distinct from score, and reporting the untested classes alongside the result · evaluating the feature rather than the model, since retrieval, prompts and tools carry most failures · a written threat model before adversarial testing · refreshing the suite from production traffic with a held-out portion · thresholds and consequences fixed in advance, plus a recorded override path · staged rollout and flag-based rollback as the answer to unimagined inputs · proportioning depth to worst single-user outcome and reversibility · and saying plainly which concerns the gate cannot retire.

Quick check

Quiz: Why can a high evaluation score be weak evidence of safety? Because the score is bounded by the suite's coverage, so a narrow suite produces a confident statement about a small part of the real failure surface.

Flashcard: What answers the risk of inputs nobody imagined? Not a deeper evaluation suite — staged rollout with live guardrails and a rollback that is a feature flag rather than a model change.