Custom Evals and LLM-as-Judge
Why every production team eventually builds its own eval set, and how to use LLM judges without being fooled by their well-documented biases.
Public benchmarks measure general capability. Your product measures whether a specific model handles your specific traffic. The correlation is loose enough that any team shipping LLMs into production ends up building its own eval set within the first three months - and once that set exists, the next question is who or what grades it.
Distribution mismatch is why public scores mislead
The model that wins MMLU may lose your support-ticket triage task because your tickets are full of product nouns, terse customer phrasing, and a long tail of edge cases the pretraining distribution under-represents. You can only discover this by evaluating on your own data. A custom eval set is not a nice-to-have; it is the precondition for any defensible model selection or regression test.
The eval-set construction recipe
- Sample real traffic. Pull 200-500 representative inputs from production logs. Stratify by intent, language, and customer segment so the tail is represented.
- Get gold answers from experts. Have your most senior domain experts write or approve the correct outputs. This is slow and expensive; do not skip it. Single-expert labels are noisy - aim for two independent labels and adjudicate disagreements.
- Categorise failure modes. Cluster the model's current errors into named buckets (hallucination, format break, refusal, off-topic, partial answer). Each bucket becomes a sub-score in the eval report.
- Bootstrap with synthetic variation. Once you have the seed set, use an LLM to generate paraphrases, edge cases, and adversarial twists - then have humans accept or reject each one. This grows the set 5-10x at low cost without losing label quality.
- Freeze the test split. Keep a held-out slice the model and prompt are never tuned against. Public benchmark contamination is now your contamination problem in miniature.
LLM-as-judge
For tasks where there is no single correct answer (open-ended generation, summarisation, dialogue), human grading does not scale. The mainstream solution is LLM-as-judge: prompt a frontier model to grade the output of the model under test.
| Framework | Approach | Notes |
|---|---|---|
| G-Eval | Chain-of-thought + form-filling, GPT-4-backed | High correlation with human (Spearman ~0.51 on summarisation) but biased toward outputs from the judge's own model family |
| Prometheus | 13B open-source judge trained on rubric-feedback pairs | Matches GPT-4 judge correlation when given a reference answer and rubric, far cheaper |
| AlpacaEval 2.0 | Pairwise preference against a fixed baseline, GPT-4 judge | 0.98 Spearman with ChatBot Arena, runs in under 3 minutes for under $10 |
| MT-Bench | Multi-turn questions scored by GPT-4 | The original LLM-as-judge methodology paper; 80%+ agreement with human preference |
Known judge biases
The MT-Bench paper catalogued the failure modes that every subsequent framework has had to mitigate:
- Position bias. Pairwise judges prefer whichever output appears first (or sometimes second) regardless of quality. Mitigate by running both orderings and averaging.
- Verbosity bias. Judges prefer longer outputs even when length adds nothing. Length-control or explicit "concise is better" instructions help, imperfectly.
- Self-preference. GPT-4 prefers GPT-4 outputs; Claude prefers Claude outputs. Cross-family evaluation reduces this, and never let the model under test also be the judge.
- Self-consistency drift. The same judge gives different scores on different days as the underlying model is updated. Pin judge model versions in your eval config.
Pairwise vs absolute scoring
Pairwise ("which of these two is better?") is easier for both humans and LLM judges - the cognitive task is comparison, not absolute calibration. Use pairwise for model selection, A/B comparison, and preference data collection. Absolute rubric scores (1-5 on each of accuracy, relevance, tone) are noisier per item but compose into a dashboard better. Production teams usually run both: pairwise for "is this regression real?" and absolute rubrics for "where exactly is the regression?"
When it falls down
- Judges miss domain mistakes. A general-purpose judge cannot tell you the SQL the model wrote is subtly wrong. For domain tasks, executable evaluation (run the code, query the database, check the answer) beats any judge.
- High agreement does not equal correctness. A judge and a human can both be confidently wrong in the same direction (length bias, tone bias). Sanity-check the eval pipeline against held-out human grading periodically.
- Cost compounds. Grading 1,000 examples with GPT-4 costs as much as your weekly experimentation budget. Pick judge model size deliberately.
Further reading
5 flashcards for this concept
Click a card to reveal the answer.