Guidelines Are the Model Specification
Why the annotation guideline determines what the model learns more than the architecture does, what a usable guideline contains, and the iteration loop that produces one.
A model learns the function its labels encode. Those labels are produced by people following a document, so that document is the actual specification of the model's behaviour, and it is typically written in an afternoon by someone who will not be doing the labelling.
Everything downstream inherits its ambiguities. A guideline that does not resolve a case produces inconsistent labels for that case, which produces a model that is inconsistent there, which surfaces months later as an unexplained behaviour that nobody can trace back to a sentence nobody wrote.
What a usable guideline contains
The decision, stated as a question. Not "label the sentiment" but "does this message express dissatisfaction with the product, as opposed to with the company, a competitor, or nothing?" Precision in the question is most of the work.
Boundary cases with rulings. The value is concentrated here. Sarcasm, mixed sentiment, quotes of someone else's opinion, messages in another language, spam: each needs a stated answer, because each will otherwise be resolved differently by different annotators.
Worked examples on both sides of each boundary, with the reasoning. Examples teach faster than rules and generalise better, and a pair straddling a line teaches the line.
An explicit escape hatch. A category for cases the guideline does not cover, plus a route for surfacing them, so ambiguity becomes visible rather than being resolved silently and inconsistently.
Priority ordering. When two rules conflict, which wins. Conflicts are common and unresolved conflicts are resolved by whoever is labelling.
The iteration loop
Guidelines are not written, they are converged on. Draft, have several people label the same fifty items, measure agreement, examine every disagreement, determine whether it was a misunderstanding or a genuine gap, revise, repeat.
Two or three rounds of this before production labelling begins is the single highest-return investment in a data project. Each round surfaces ambiguities that would otherwise appear as label noise, and the disagreements are the diagnostic: a case where careful people disagree is a case the guideline does not cover.
When it breaks
Guidelines written by people who never label. The author does not encounter the cases that make the task hard, so the document covers the easy ninety percent. Anyone writing a guideline should label a few hundred items first.
Growth without structure. A guideline that accretes special cases becomes a document nobody can hold in mind, and annotators fall back on intuition, which is where it started. Periodic restructuring around principles, with cases as illustrations, is what keeps it usable.
Changing guidelines mid-project. A revision at week six means the first six weeks' labels follow different rules, and unless the change is recorded and the affected batches identified, the dataset contains two definitions of the label. Versioning the guideline and stamping each label with the version is the fix.
The guideline is not shipped with the data. A dataset without its labelling guideline is a dataset whose labels cannot be interpreted. It belongs in the datasheet and in the repository, not in a document the annotation vendor holds.
12 flashcards for this concept
Click a card to reveal the answer.