Eval-Driven Development

The typical LLM feature ships on vibes. Someone tries twenty prompts, the outputs look good, it goes live, and then the team spends six months making changes they cannot evaluate. Every prompt edit is a coin flip: it fixes the complaint that prompted it and silently breaks something nobody is checking.

Eval-driven development is the boring fix. Before changing the prompt, write down what "better" means and how you would detect it. It is test-driven development with a probabilistic system under test, and the differences from TDD are the interesting part.

Start from failures, not from coverage

The instinct is to build a broad representative test set. That is the wrong first move, because a representative set is mostly cases the system already handles, and it will show a flat 90 percent that never moves.

Start instead with a failure-derived set: every production complaint, every escalation, every output a reviewer flagged. Fifty of these are worth a thousand random samples, because each one is a case the system is known to get wrong, so the metric has room to move. Add the passing cases later, as regression protection.

The practical loop:

1. Collect traces from production, including the full prompt, tools called, and output.
2. Have a human label a sample, with a written rubric, not a gut score.
3. Turn each distinct failure into a test case with an expected property.
4. Automate the check: a string assertion, a schema validator, a unit test, or, only where nothing cheaper works, an LLM judge.
5. Run the suite on every prompt, model, or retrieval change.

Choose the cheapest verifier that works

There is a hierarchy, and teams reach for the top of it far too early.

An LLM judge is a model with its own error rate, and that error rate must be measured against human labels before its numbers are trusted; see custom evals and LLM judges. A judge with 85 percent agreement with humans cannot resolve a 3-point difference.

Keeping the set honest

Three disciplines separate a useful eval set from decorative CI.

Hold out a slice you never look at. Iterating against the whole set is training on the test set by hand, and it happens within weeks.

Version the set with the product. When the intended behaviour changes, cases become wrong. A stale case that now fails correctly is worse than no case, because the team learns to ignore red.

Track per-case results, not just the aggregate. A stable 82 percent can hide ten cases fixed and ten broken. The diff of which cases flipped is the actual signal, and aggregate scores conceal it.

When it breaks

Eval sets rot faster than test suites. Model upgrades, prompt rewrites and product changes all invalidate cases. Budget maintenance time or the suite becomes a source of ignored failures.

Small sets cannot resolve small changes. Fifty cases give a standard error around 6 points at 80 percent accuracy. That set can catch a regression; it cannot adjudicate a 2-point improvement. See error bars for evals for the sizing arithmetic.

Offline agreement does not guarantee online improvement. The eval measures the model's output. The product measures whether users got what they came for. Keep at least one online metric, whether task completion, escalation rate, or thumbs-down rate, wired to the same release decision, because a change that improves every offline case and raises the escalation rate is a regression.