Inter-Annotator Agreement and What It Bounds
Why raw agreement overstates reliability, what Cohen's and Krippendorff's coefficients correct for, and why agreement is the ceiling on any model trained from those labels.
If two careful annotators following the same guideline agree on 70 percent of items, a model trained on those labels cannot meaningfully exceed 70 percent, because the remaining 30 percent has no consistent answer to learn. Agreement is therefore not a data quality statistic; it is a bound on achievable performance, and measuring it before training is what tells you whether the target is reachable.
Why raw agreement overstates
Two annotators labelling a binary task where 90 percent of items are negative will agree about 82 percent of the time by chance alone. Raw agreement is dominated by the base rate and says almost nothing about whether they are applying the same criterion.
Cohen's kappa corrects for chance agreement between two annotators:
where \(p_o\) is observed agreement and \(p_e\) is expected agreement under independence. It ranges up to 1, with 0 meaning chance-level.
Krippendorff's alpha generalises this to any number of annotators, handles missing data, and supports ordinal, interval and ratio scales through a distance function, which matters because disagreeing between "3" and "4" on a five-point scale is not the same as disagreeing between "1" and "5".
Conventional interpretation puts 0.8 and above as good, 0.67 to 0.8 as tentative, and below 0.67 as unreliable for drawing conclusions, though these thresholds are conventions from content analysis rather than results, and the acceptable level depends on the task.
The paradoxes
Kappa is depressed by severe class imbalance even when agreement is genuinely high, because \(p_e\) approaches \(p_o\). A task with 98 percent negatives can show near-perfect agreement and a kappa near zero. Reporting kappa alone on an imbalanced task is misleading in the pessimistic direction, and the prevalence and bias figures alongside it are what make it interpretable.
What low agreement means
It is not automatically a problem to fix. Three possibilities need distinguishing.
The guideline is ambiguous, which is fixable and is the most common cause. The task is genuinely subjective, in which case the disagreement is the signal and aggregating to a single label destroys information. Or the annotators are undertrained or inattentive, which is a process problem.
For subjective tasks, keeping the distribution of labels rather than a majority vote preserves what the disagreement encodes, and training on soft labels frequently outperforms training on the aggregated hard label.
When it breaks
Agreement is measured once and assumed stable. Annotator understanding drifts, new people join, and the input distribution changes. Continuous measurement on an overlapping subset, typically five to ten percent of items double-labelled throughout, is what catches drift.
Majority vote is treated as truth. Three annotators agreeing establishes consistency, not correctness. Where all three share a misunderstanding, the aggregate is confidently wrong, and only expert adjudication on a sample detects it.
Aggregation discards the hard cases. The items with disagreement are the informative ones, and majority voting flattens them into confident labels that teach the model that a genuinely ambiguous case has a clear answer.
Agreement is reported without the guideline version. A kappa computed under one guideline says nothing about labels produced under another, so the two must be reported together.
12 flashcards for this concept
Click a card to reveal the answer.