Position Bias and the Examination Hypothesis

Rank 1 receives several times the clicks of rank 5 for the same document. Move a document from rank 5 to rank 1 and its click-through rate rises without its relevance changing at all. Any model trained on raw click-through rate therefore learns position as much as it learns quality, and since the current ranker determined position, the model learns to imitate the current ranker.

The feedback loop closes: the ranker determines which documents get seen, seen documents get clicked, clicks become training labels, and the retrained ranker reinforces its predecessor's choices. A genuinely better document sitting at rank 20 never accumulates the evidence that would promote it.

The examination hypothesis

The standard model factorises a click into two independent events: the user examined the position, and the document was relevant.

\(P(E=1\mid k) = p_k\) is the propensity, depending only on rank. Under this model, the observed click rate is a biased estimate of relevance with a known multiplicative distortion, and dividing by \(p_k\) recovers an unbiased estimate.

The hypothesis is a simplification with known defects. Examination in reality depends on the snippet, on what appeared above, and on whether the user found an answer earlier. Its value is that it is wrong in bounded, arguable ways, and that it makes the correction tractable.

Estimating propensities

Result randomisation swaps a result to a random position for a small traffic slice, so the same document is observed across ranks and the click ratio identifies \(p_k\) directly. It is the cleanest method and it degrades user experience, which limits how much traffic can carry it.

Swap interventions are the gentler version: swap adjacent pairs, or swap only within the top few positions, sacrificing far less quality for a noisier estimate.

Intervention harvesting exploits variation that already exists. When a ranker is updated or an A/B test runs, the same document appears at different ranks across the logs, and that natural variation identifies propensities without any deliberate degradation. It requires the logs to record which ranker served each impression.

EM-based joint estimation infers propensities and relevance together from click logs alone, under the examination model. It requires no intervention and its identification is weaker, since it leans entirely on the model being correct.

The other click biases

Position bias is the largest and not the only one. Trust bias: users click top results even when the snippet is unconvincing, so the top position inflates clicks beyond examination alone. Presentation bias: rich snippets, images and answer boxes change click rates independent of relevance. Selection bias: unranked documents receive no impressions at all, so they generate no data of any kind, which is a harder problem than reweighting because there is nothing to reweight.

When it breaks

Propensity correction inflates variance. Dividing by \(p_{10} = 0.05\) multiplies that observation's weight by 20, so rare deep clicks dominate the gradient. Propensity clipping is standard and reintroduces bias, which is the same trade seen in every importance-weighted estimator.

A wrong propensity model produces a confidently wrong correction. If examination genuinely depends on the snippet or on what was above, a rank-only propensity mis-corrects in a way that is invisible in the training metrics, and the corrected model can be worse than the uncorrected one.

Zero-impression documents cannot be recovered. The correction handles documents that were seen with varying probability. Documents the candidate generator never surfaced have propensity zero, and no weighting resurrects them; only exploration does.

Offline and online improvements can diverge. A debiased model reorders results in ways that change user behaviour, and the propensities estimated under the old ranking no longer describe the new one. Propensity estimation is an ongoing process, not a one-off calibration.