A platform ships AI features and cannot tell whether changes improve or degrade quality. What evaluation infrastructure is required, and in what order?

In order

1. A held-out set of real tasks with known-good outputs. Drawn from actual usage, not invented. This is the foundation and the step teams skip because it is unglamorous and requires judgement to build.

It should include the hard cases, the ambiguous cases, and the ones that previously failed — a set of easy cases measures nothing.

2. A comparison harness. Run a candidate configuration — model, prompt, retrieval settings — against the set and report differences from the current baseline. Automation matters: an evaluation requiring manual review will be run rarely and skipped under pressure.

3. Metrics appropriate to the task. Exact match where there is a correct answer; retrieval metrics for whether the right source was surfaced; rubric-based scoring for open-ended outputs. Score the components separately — retrieval and generation quality are different problems with different fixes, and a single end-to-end score cannot tell you which one regressed.

4. Regression gating in the release path, so a change that degrades the evaluation set does not ship silently.

5. Online measurement. User-visible outcomes — task completion, edit rate on generated content, escalation rate, thumbs-down rate. Offline evaluation correlates imperfectly with user outcomes, so the online signal is the final arbiter.

6. Feedback capture that feeds the evaluation set, so real failures become permanent test cases. This is what makes the harness improve rather than decay.

What makes evaluation hard here

Outputs are non-deterministic, so a single run is noise. Evaluate across multiple samples and compare distributions rather than instances.

Quality is often subjective, which is why rubric-based scoring with a defined rubric — applied consistently — beats an undefined notion of "better".

The evaluation set decays. As the product and corpus change, the set stops representing real usage. Maintaining it is ongoing work, and an unmaintained set produces confident wrong conclusions.

The consequence of not having it

Every prompt, model and retrieval change is an unmeasured risk. Teams ship on impressions, discover regressions through user complaints, and cannot distinguish improvement from noise.

It is also the prerequisite for model selection being an ongoing decision rather than a historical one — without a harness, a platform runs whatever it chose originally, indefinitely.