LLM-as-Judge
Using a model to score another model's output against a rubric, providing scalable evaluation where exact-match assertions do not apply.
Most LLM outputs cannot be checked by string comparison — there are many acceptable answers. Human review is accurate and does not scale. Model-based scoring sits between: cheap enough to run continuously, accurate enough to detect regressions.
What makes it reliable enough to act on:
A specific rubric with defined criteria, not "is this good?". Score faithfulness to the retrieved sources, completeness against the question, adherence to format, and absence of prohibited content — each separately, because a single aggregate score hides which dimension moved.
Calibration against human judgement on a sample. Without it the scores are unanchored, and a judge that systematically disagrees with humans is worse than no evaluation because it is trusted.
A capable judge model, usually stronger than the one being evaluated.
Awareness of the known biases: judges favour longer answers, favour outputs from the same model family, and are sensitive to position when comparing options. Randomising order and controlling for length mitigates these.
The evaluation layers it fits into: deterministic checks first (schema, required fields, forbidden content) because they are free and unambiguous; retrieval metrics separately from generation metrics, since a bad answer may be either; model-based scoring for quality; and human review on a sample, which remains the ground truth everything else is calibrated against.