LLM-Assisted Annotation and Label Verification

A 2023 study gave 2,382 tweets to ChatGPT and to Mechanical Turk workers and scored both against trained annotators on tasks such as relevance, stance, topic and frame detection. Zero-shot ChatGPT beat the crowd on four of five tasks, had higher intercoder agreement than both crowd workers and trained annotators, and cost under $0.003 per annotation, about twenty times less than MTurk (Gilardi, Alizadeh & Kubli, 2023, ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks, PNAS 120(30), arXiv:2303.15056). The question soon became how to use model labels without corrupting what is built on them.

The neighbouring concepts on label noise and inter-annotator agreement describe what imperfect labels do to training. This concept is about the design of a pipeline in which most labels come from a model and humans are spent where they buy the most validity.

What the evidence actually supports

The follow-up literature narrowed the headline rather than overturning it. Pangakis, Wolken and Fasching replicated 27 annotation tasks from 11 published social-science datasets with GPT-4 and found performance promising but highly contingent on the dataset and task type, strong on some and poor on others, which cannot be known without human labels for that task (Pangakis, Wolken & Fasching, 2023, Automated Annotation with Generative AI Requires Validation, arXiv:2306.00176). Reiss showed that small prompt rewordings and repeated runs of identical inputs changed ChatGPT's classifications enough to fall below conventional reliability thresholds (Reiss, 2023, Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark, arXiv:2304.11085).

The human baseline moved too. One study estimated that 33 to 46% of MTurk workers on a summarisation task used LLMs to do it (Veselovsky, Ribeiro & West, 2023, Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks, arXiv:2306.07899), so "LLM versus crowd" comparisons may partly compare a model with itself. Whether model annotation is a substitute for crowd work or only for its worst tier remains disputed, and the answer is task-specific.

Accuracy is not validity

Suppose the goal is not to train a classifier but to estimate a quantity, such as the share of complaints mentioning billing. A labeller that is 90% accurate but errs systematically in one direction shifts that estimate, and treating its labels as ground truth yields confidence intervals that are narrow and wrong. Egami et al. showed that directly using surrogate labels leads to substantial bias and invalid intervals even at 80 to 90% surrogate accuracy, and proposed a doubly robust estimator that combines surrogates with a random gold-standard subsample (Egami et al., 2023, Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models, NeurIPS, arXiv:2306.04746).

Prediction-powered inference gives the simplest form of that idea for a mean (Angelopoulos et al., 2023, Prediction-Powered Inference, Science 382(6671)). With \(N\) model-labelled items \(\hat Y'_j\) and a random sample of \(n\) items carrying both model labels \(\hat Y_i\) and human labels \(Y_i\),

The second term, the rectifier, measures the model's bias on the audited sample and subtracts it.

A worked case, with illustrative numbers. The model labels 100,000 complaints and flags 30% as billing-related. Humans label a random 1,000, on which the model also says 30% but humans say 24%, so the rectifier is \(0.06\) and the corrected estimate is \(0.24\). If the model and humans disagree on 10% of audited items, \(\hat\sigma^2_{\hat Y - Y} \approx 0.10 - 0.06^2 = 0.096\); with \(\hat\sigma^2_{\hat Y} = 0.21\), the standard error is \(\sqrt{0.21/100{,}000 + 0.096/1{,}000} \approx 0.0099\), a 95% interval of about \(\pm 1.9\) points. The same 1,000 human labels alone give \(\sqrt{0.24 \cdot 0.76/1{,}000} \approx 0.0135\), or \(\pm 2.6\) points. The model labels did not replace the humans; they made the human sample worth more.

Routing by confidence

For building datasets rather than estimates, the common design routes items: accept model labels above a confidence or verifier-score threshold \(\tau\) and send the rest to people. Wang et al. found model labels cost 50 to 96% less than human labels for equivalent downstream performance, and mixing the two did better still under a fixed budget (Wang et al., 2021, Want To Reduce Labeling Cost? GPT-3 Can Help, Findings of EMNLP, arXiv:2108.13487). A later CHI study built a verifier over inputs, labels and model explanations to pick which items humans re-annotate, and found that showing annotators the model's label raised their accuracy when the model was right and lowered it when the model was wrong (Wang et al., 2024, Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels, CHI). That is the same anchoring that explanations produce in decision support.

When it breaks

Routing destroys randomness. Items sent to humans are the low-confidence ones, so their disagreement rate says nothing about errors among the accepted, confident items. Estimating the error of what was auto-accepted needs a separate random audit of that stratum, and PPI's guarantee requires a random gold sample, not the routed one.

Confidence is not calibrated by default. A threshold set on one prompt and model version drifts when either changes, and confident systematic errors are exactly the ones routing never surfaces.

Blind review becomes rubber-stamping. Showing the model label speeds review and imports its errors. Collecting the human label before revealing the model's preserves independence at a cost in time.

Circularity. Labels from one model family used to train or evaluate the same family reproduce its blind spots, which is why LLM judges used for evaluation carry the same audit burden as LLM annotators used for training.