A team ships an LLM feature and cannot tell whether changes improve it. What evaluation infrastructure is needed, and what does it not solve?

What is needed

• A golden set: representative inputs with known good outputs or with defined criteria. It must include the failure cases discovered in production, which is what turns it from a benchmark into a regression suite.
• Automated scoring for what can be scored automatically — exact match, structured output validity, presence of required elements, absence of forbidden content, retrieval correctness.
• Model-based judging for what cannot, with the judge itself validated against human ratings. An unvalidated judge measures the judge's preferences, and its agreement with humans should be measured periodically rather than assumed.
• Human evaluation on a sample, which remains the ground truth and calibrates everything else.
• Run on every change to prompts, models, retrieval or chunking — because improvements in one area silently regress another, and that is the normal state of systems without evaluation.
• Production feedback captured and fed back: thumbs, corrections, escalations, abandonment. This is the cheapest signal available and the most often uncollected.

What it does not solve

• Coverage. A golden set represents what you thought to include, and production contains inputs nobody anticipated. It catches regressions well and novel failures not at all.
• Subtle quality. Whether an explanation is genuinely pedagogically useful is not measurable by any automated proxy, and for an education product that is the thing that matters most.
• Distribution shift. Real usage changes and a static evaluation set ages, so it needs periodic refresh from production traffic.
• The provider changing under you. A model updated without notice can change behaviour, which is why the harness must be runnable on demand and why version pinning matters where the provider offers it.

The organisational requirement

Evaluation must be cheap enough to run on every change, or it will be run before releases and then not at all. That means fast, automated, and integrated into the pipeline rather than being a separate exercise somebody schedules.

A team that cannot answer "did that prompt change make things worse" within an hour is developing by intuition, and intuition is unreliable in a system where fluent output is uncorrelated with correctness.