LLM Relevance Judgments

Two groups of trained TREC assessors, labelling the same ad hoc documents independently, agree at Cohen's \(\kappa\) of about 0.52. A GPT-4 prompt tuned against TREC Robust 2004 labels reached \(\kappa = 0.64\) on a stratified sample of 3,000 query-document pairs, and 42 paraphrases of that prompt ranged from 0.50 to 0.72 (Thomas, Spielman, Craswell and Mitra, 2024, Large Language Models can Accurately Predict Searcher Preferences, SIGIR, arXiv:2309.10621). The same paper reports that Bing uses LLM labels for relevance at scale. The judging bottleneck that has shaped IR evaluation since Cranfield looks, on those numbers, to be over. The rest of this concept is about why many of the people who run evaluations disagree.

The general biases of LLM judges, such as position, verbosity and self-preference, are covered in the concept on custom evals and LLM judges. This concept is about the IR-specific use: replacing the qrels in a test collection.

How the labels are produced and checked

The judge receives the query, optionally a description and narrative of the information need, the document text, and a graded scale such as 0 (not relevant), 1 (relevant) and 2 (highly relevant). Agreement with human labels is measured with Cohen's kappa,

where \(p_o\) is observed agreement and \(p_e\) is agreement expected by chance from each labeller's marginal label rates. If judge and assessor agree on 85% of binary labels and chance agreement is 55%, then \(\kappa = 0.30 / 0.45 \approx 0.67\). Kappa at the document level is only one lens. What an evaluation needs is that systems come out in the same order, usually measured with Kendall's \(\tau\) between system rankings under human and machine qrels.

Prompt design moved the numbers far more than intuition predicts. In Thomas et al.'s factorial study, asking the model to rate aspects (topicality and trustworthiness) before an overall label raised \(\kappa\) by 0.21 on average, while asking it to simulate five judges lowered it by 0.13 and adding a "search quality rater" role lowered it by 0.04. Their method, which is the part most often skipped when the paper is cited, selects the model and prompt against a small set of gold labels from real searchers rather than trusting any prompt out of the box.

What it is safe for

Faggioli and colleagues framed the question as a spectrum running from humans judging alone, through machine assistance, to fully automatic judgment, and set out the risks at the automatic end before much evidence existed (Faggioli et al., 2023, Perspectives on Large Language Models for Relevance Judgment, ICTIR, arXiv:2304.09161). The strongest evidence for the automatic end came from the TREC 2024 RAG track: across 77 runs from 19 teams, system rankings from UMBRELA's fully automatic LLM judgments correlated highly with rankings from fully manual assessment, adding humans to the loop did not raise that correlation, and human assessors were stricter than the LLM (Upadhyay et al., 2024, A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look, arXiv:2411.08275).

Leniency matters more than it sounds. Alaofi and colleagues found LLM judges more likely than humans to call a passage relevant, and showed they could be fooled by passages stuffed with query words or carrying an instruction asserting relevance, so an LLM's non-relevant labels are more trustworthy than its relevant ones (Alaofi, Thomas, Scholer and Sanderson, 2024, LLMs can be Fooled into Labelling a Document as Relevant, SIGIR-AP).

The circularity problem

Clarke and Dietz replied directly to the TREC 2024 claim. They built a system designed to exploit the automatic judge and showed it scored higher without retrieving better. They also showed that if participating systems adopted the judge's own model as a final-stage reranker, the system ranking produced by LLM qrels drifted away from the human one (Clarke and Dietz, 2024, LLM-based Relevance Assessment Still Can't Replace Human Relevance Assessment, arXiv:2412.17156). The mechanism is simple. If judge \(J\) and ranker \(f\) share a model, then optimising \(f\) against qrels from \(J\) rewards agreement with \(J\), and the metric stops measuring agreement with users. Soboroff, whose group ran TREC for years, put the position bluntly in the title of a keynote: Don't Use LLMs to Make Relevance Judgments (Soboroff, 2024, arXiv:2409.15133).

The positions are closer than the titles suggest. Both sides accept LLM labels for development-time iteration and for triaging what humans should judge. The disagreement is over test collections meant to rank future systems, where the judge's preferences become a target.

When it breaks

A prompt is a sample from a wide distribution. The 0.50 to 0.72 paraphrase spread came from wording changes a reader could not rank by eye. Freeze the prompt, model version and decoding settings, and re-validate against gold labels when any of them changes.

Retired models make labels unreproducible. Qrels produced by an API model that is later withdrawn cannot be regenerated, so the collection cannot be extended with consistent labels.

System-level correlation hides top-of-leaderboard errors. A high Kendall's \(\tau\) over 77 runs can coexist with swaps among the top five, which are the only comparisons most readers care about.

Documents can be written for the judge. Once a judge is known, content can include text that persuades the model, and the leniency bias does the rest.

Gold labels are still required. Every successful deployment reported so far selected and monitored the judge against human labels from real searchers, so an LLM judge reduces how many human labels you need. It does not remove the need for them.