Preference Data Collection

Alignment training runs on human preferences: shown two model outputs, a person picks the better one. The choice of comparison rather than rating is deliberate and well founded, and everything downstream, the reward model, the policy, the model's character, is shaped by details of how those comparisons were collected.

Why comparison beats rating

People are unreliable at absolute judgement and reliable at relative judgement. Asked to rate a response from 1 to 7, different annotators use different parts of the scale, the same annotator drifts across a session, and the numbers are not comparable across people. Asked which of two is better, the judgement is direct and the scale problem disappears.

Comparisons also compose. The Bradley-Terry model converts pairwise outcomes into a latent quality score, which is what a reward model is fitted to, so a set of local judgements yields a global ordering without anyone having to be calibrated.

The biases that get learned

Whatever correlates with the chosen response in the data becomes part of the reward, and several things correlate that nobody intended.

Length. Longer responses are preferred at a rate not explained by quality, and this is the best-documented bias in the area. A reward model trained without correction learns that longer is better, and the policy optimising against it produces verbosity. Length-controlled evaluation and explicit length penalties exist because the effect is large enough to dominate.

Confidence and assertiveness. Hedged answers are rated lower than confident ones regardless of whether the confidence is warranted, which is one route by which models become overconfident.

Agreement with the rater's view, which is the mechanism behind sycophancy.

Formatting. Structure, headers and lists are preferred over equivalent prose, which is why models trained this way produce heavily formatted output whether or not the task calls for it.

Position. In an interface where one response is always shown first, position affects choice, so randomising order per item is mandatory rather than good practice.

Design choices that matter

Who the raters are determines whose preferences the model learns. A model trained on the preferences of one demographic, one language community, or one contractor pool has learned those preferences and will present them as neutral.

What they are asked matters as much. "Which is better" invites whatever the rater values; "which is more accurate" or "which is more helpful for someone trying to X" produces a different and more specific signal. Multi-dimensional collection, rating helpfulness and harmlessness and accuracy separately, gives a reward model more to work with and costs more per item.

How disagreement is handled. Preference data has genuine disagreement, and averaging it produces a reward model targeting a consensus nobody holds. Retaining rater identity permits modelling the distribution rather than its mean.

When it breaks

Rater expertise bounds the signal. A non-expert cannot judge which of two technical answers is more accurate, so on expert tasks preference data measures presentation. This is a fundamental limit on what preference training can teach, and it is why verifiable-reward methods are used where correctness can be checked.

Reward models are overoptimised. A policy pushed hard against a reward model finds regions where the model is wrong, producing outputs that score highly and are bad. The KL penalty against the reference policy exists to bound this, and the right strength is an empirical question.

Preference data ages. As the policy improves, the comparisons in an old dataset are between outputs the current model no longer produces, so the reward model is calibrated on an obsolete distribution. Iterative collection against the current policy is what keeps it relevant.

Annotator welfare is a real cost. Preference and safety annotation involves exposure to harmful content, and the labour conditions of the workforce producing alignment data are a documented concern that belongs in the discussion rather than outside it.