Human Data & Annotation
Guideline design, inter-annotator agreement, preference collection, rater sourcing and label noise.
7concepts
74flashcards
51minutes of reading
- 01 Active Learning and Annotation Budgets How choosing which examples to label, by uncertainty or by coverage, can cut annotation cost, why cold starts and sampling bias undermine it, and why the gains that looked large for models trained from scratch often shrink or vanish with pretrained ones.
- 02 Guidelines Are the Model Specification Why the annotation guideline determines what the model learns more than the architecture does, what a usable guideline contains, and the iteration loop that produces one.
- 03 Inter-Annotator Agreement and What It Bounds Why raw agreement overstates reliability, what Cohen's and Krippendorff's coefficients correct for, and why agreement is the ceiling on any model trained from those labels.
- 04 LLM-Assisted Annotation and Label Verification What the evidence says about language models as annotators, why a labeller that is 90% accurate can still produce biased conclusions, and the audit and routing designs that combine cheap model labels with a small human sample into statistically valid results.
- 05 Label Noise and Learning Through It How random and systematic label noise differ in their effect on a model, why memorisation of noisy labels happens late in training, and the techniques that find mislabelled data cheaply.
- 06 Preference Data Collection Why pairwise comparison replaced absolute rating for alignment data, the biases that contaminate it, and the design choices that determine what a reward model actually learns.
- 07 Rater Sourcing, Quality and Welfare The tradeoffs between crowd, vendor, expert and internal annotation, the mechanisms that maintain quality, and the working conditions that shape both the data and the ethics.