Active Learning and Annotation Budgets

In 1994 Lewis and Gale reported that letting a classifier choose which newswire stories to have labelled, by picking the ones it was least sure about, reduced the labelling needed to reach a given effectiveness by as much as 500-fold (Lewis & Gale, 1994, A Sequential Algorithm for Training Text Classifiers, SIGIR, arXiv:cmp-lg/9407020). Twenty-five years later, Lowell, Lipton and Wallace ran active learning across a range of NLP models and tasks and concluded that its benefits did not generalise reliably, and that a dataset acquired with one model often did no better than random sampling when used to train another (Lowell, Lipton & Wallace, 2019, Practical Obstacles to Deploying Active Learning, EMNLP, arXiv:1807.04801). Both results are real. The distance between them is the practical content of this topic.

Choosing what to label

Active learning runs a loop: train on the labelled set \(L\), score every item in the unlabelled pool \(U\) with an acquisition function, send the top \(b\) to annotators, add them to \(L\), retrain. With model predictive distribution \(p_\theta(y \mid x)\) over classes \(y\), the classic uncertainty scores are

least confidence, margin between the top two classes \(\hat y_1, \hat y_2\) (select the smallest), and predictive entropy. All three pick points near the current decision boundary.

Uncertainty sampling in batches picks near-duplicates: fifty ambiguous examples from the same corner of input space teach the model one thing fifty times. The core-set approach replaces uncertainty with coverage. Given a feature embedding and distance \(\Delta\), choose the batch \(s\) to minimise the largest distance from any pool point to its nearest labelled point,

the \(k\)-center problem, which is NP-hard but has a greedy 2-approximation (Sener & Savarese, 2018, Active Learning for Convolutional Neural Networks: A Core-Set Approach, ICLR, arXiv:1708.00489). Their bound ties the gap between loss on the labelled set and loss on the whole pool to that covering radius, which is why diversity, not uncertainty, is the quantity it targets.

The budget arithmetic

Consider a pool of 200,000 support tickets, an assumed labelling cost of $0.40 per item and a $4,000 budget: 10,000 labels. Spent at random, that is a 5% sample. Spent actively in ten rounds of 1,000, it requires ten retraining and rescoring passes over 200,000 items, and annotators wait while each runs. Smaller batches make selection more adaptive and multiply both costs; one large batch is operationally simple and barely active at all. The real budget is therefore labels plus compute plus annotator idle time plus engineering, and active learning only pays when its label saving exceeds the other three.

Cold starts and sampling bias

The loop needs a model to score uncertainty, and the first model is trained on almost nothing, so its uncertainty is noise. Hacohen, Dekel and Weinshall showed that the best strategy flips with budget: at low budgets, querying typical examples wins, while uncertain or unrepresentative examples win only once the budget is large. Their low-budget method, with semi-supervised training, reached 93.2% accuracy on CIFAR-10 from ten labelled examples, an improvement the authors report as 39.4% over random selection (Hacohen, Dekel & Weinshall, 2022, Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets, ICML, arXiv:2202.02794).

The deeper problem is sampling bias. As queries concentrate near the model's current boundary, the labelled set stops resembling the data distribution, and Dasgupta and Hsu showed that some such heuristics are not statistically consistent: even with unlimited labels they can fail to converge to a good classifier, because a region the early model was wrongly confident about is never queried (Dasgupta & Hsu, 2008, Hierarchical Sampling for Active Learning, ICML). The labelled set is also useless for estimating accuracy, since it is deliberately unrepresentative; evaluation needs a separate random sample.

Do pretrained models make it obsolete?

Here the literature genuinely disagrees. Chan, Li and Oymak combined self-supervised pretraining and semi-supervised training on CIFAR-10 and CIFAR-100 and found no additional benefit from state-of-the-art active learning on top (Chan, Li & Oymak, 2020, On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?, arXiv:2011.08121). Against that, Ein-Dor et al. found active learning did improve BERT classifiers under small budgets and class imbalance, especially when the initial labelled set came from keyword queries and was therefore a biased sample of the minority class (Ein-Dor et al., 2020, Active Learning for BERT: An Empirical Study, EMNLP). One reading consistent with both is that pretraining removes most of the gain on balanced benchmarks, where random sampling already covers the space, and leaves it where random sampling fails: rare classes, where a 1% positive rate means a random 1,000 labels contain about ten positives.

When it breaks

The data outlives the model. Labels selected for today's model are a biased sample for tomorrow's, which is Lowell et al.'s transfer result. If the dataset is the long-lived asset, random or stratified sampling protects its reuse.

Uncertainty finds garbage. The items a model is least sure about include corrupted, off-topic and genuinely ambiguous inputs that no label will fix, and annotators spend the budget on them.

Retrospective simulations flatter. Many published curves simulate annotation on already-labelled datasets, with no annotator latency, disagreement or fatigue on hard items, all of which active selection concentrates.

Stopping is undefined. Without a representative held-out set, there is no honest signal that the next round is worth buying.