LLM Regression Testing in CI

A team adds an LLM evaluation step to its pipeline: 400 golden cases, the candidate must not score more than two points below the main branch. Within a week engineers are re-running the job until it passes. Nothing is wrong with the code under test. The gate is failing roughly one pull request in thirteen on noise alone, and a gate that fails randomly teaches people to ignore it.

Data-dependent tests and behavioural suites covers what to test: capability probes, invariance and directional cases, failures turned into permanent tests. This concept covers the harder operational question of how to run such a suite as a merge gate on a system whose outputs are random, whose grader is itself a model, and whose every test costs money.

Three sources of noise

Case sampling. A golden set is a sample of the behaviour you care about. With \(n\) cases and pass rate \(p\), the standard error of the pass rate is \(\sqrt{p(1-p)/n}\). At \(n = 200\) and \(p = 0.9\) that is 2.1 points, so a 95 percent interval spans about ±4 points, wider than most regressions worth catching.

Generation. Outputs vary between calls even at temperature zero. Sampling 1,000 completions of one prompt at temperature 0 from Qwen3-235B produced 80 distinct outputs, which Horace He traced to kernels that are not batch-invariant, so results depend on server load (He, 2025, Defeating Nondeterminism in LLM Inference, Thinking Machines Lab). A hosted endpoint gives you no control over this.

Grading. Model judges agree with human preferences at over 80 percent in MT-Bench, similar to human-human agreement, and show position, verbosity and self-enhancement biases (Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS Datasets and Benchmarks, arXiv:2306.05685). A judge that flips on borderline cases adds variance on top of generation variance.

Compare pairs, not scores

Scoring the candidate and the baseline independently and subtracting throws away the most useful structure: both ran on the same cases. Treating evaluation questions as draws from a super-population, Miller recommends paired differences and clustered standard errors when cases share structure (Miller, 2024, Adding Error Bars to Evals, arXiv:2411.00640).

For pass/fail cases, let \(p_{10}\) be the share of cases the baseline passes and the candidate fails, and \(p_{01}\) the reverse. The estimated change is \(\hat\Delta = p_{01} - p_{10}\), with

Only discordant cases contribute. With 400 cases and 4 percent discordant in each direction, the standard error is \(\sqrt{0.08/400} \approx 1.4\) points. A rule that fails the build when \(\hat\Delta < -2\) points fires on a truly unchanged system with probability \(\Phi(-2/1.4) \approx 7.6\) percent, which is the team from the opening. At 20 pull requests a day, that is one or two false failures daily.

The fixes are statistical, not cultural. Widen the threshold to about three standard errors, or increase \(n\), or cut discordance by sampling each case \(k\) times and scoring the majority, which reduces generation noise at \(k\) times the cost. Where the budget cannot support that, make the gate asymmetric: fail only on large drops, and route smaller ones to a human reviewer with the discordant cases attached.

The cost budget decides the architecture

Every run is inference plus grading. Using illustrative prices of $3 per million input tokens and $15 per million output tokens, 300 cases sampled 3 times with 2,000 input and 500 output tokens each cost \(900 \times \$0.0135 \approx \$12\). A judge call per sample reading 3,000 tokens and writing 200 adds about $11. At roughly $23 a run, 40 runs a day over 22 working days is over $20,000 a month.

That number pushes teams into tiers. Every pull request gets cheap deterministic checks: schema validity, tool-call parsing, refusal detection, exact-match cases, a small smoke set. The full paired statistical suite runs when the diff touches a prompt, model identifier, retrieval configuration or decoding parameter, and nightly on the main branch so drift from provider changes is caught even without a diff. Caching responses keyed on the full version manifest and case ID means unchanged components are not re-billed.

When it breaks

The judge is part of the test. Changing the judge model, its prompt or its version changes every score, so a baseline graded by the old judge cannot be compared with a candidate graded by the new one. Pin the judge and re-grade both arms after changing it.

Golden sets overfit. A set that gates every merge becomes the target prompts are tuned against, and passing it stops predicting production quality. Rotating in cases from recent traces keeps it honest and breaks longitudinal comparability, so keep a frozen core alongside.

Blocking versus advisory is contested. Some teams argue LLM evals are too noisy to block merges and should only report; others argue an advisory check is ignored. A defensible middle is blocking on deterministic checks and large statistically clear regressions, advisory on everything else.

Averages hide the regression that matters. A net-zero \(\hat\Delta\) can conceal 12 newly failing safety cases offset by 12 newly passing formatting cases. Critical-case subsets need their own zero-tolerance assertions.