Reading Its Own Handwriting: Position Bias, Feedback Loops and Counterfactual Correction

Take a single search result and move it from rank 5 to rank 1. Its click-through rate rises several-fold. Nothing about the document changed, nothing about its relevance changed, and the only thing that changed was how likely a user was to look at it.

Now train a ranker on click-through rate. It learns that documents at rank 1 are good, and rank 1 was chosen by the previous ranker. The new model is not learning relevance; it is learning to reproduce the ranking that generated the training data, with a confidence that increases at every retraining cycle. A genuinely better document sitting at rank 20 never accumulates the evidence that would promote it, because nobody looked at it.

This is the defining structural problem of any system that both chooses what people see and learns from what they do. It affects search, recommendation, feeds, ad selection, and increasingly the retrieval layer of production LLM systems. The fix is not a better loss function, and it is not more data. It is a counterfactual argument about what would have happened under a different policy, and it requires the system to have been deliberately built to make that argument possible.

Why this matters: A ranker trained on raw logged clicks reliably converges toward its own predecessor. The offline metrics improve, the online metrics do not, and the diagnosis is invisible in any dashboard because the model is doing exactly what the objective asked. Fixing it requires logging decisions made before the first model shipped, which is why most systems cannot fix it retroactively.

TL;DR

• Clicks are relevance multiplied by the probability the user looked. The examination hypothesis factorises this as \(P(C) = P(E \mid k)\cdot P(R \mid d)\), and dividing by the rank-dependent examination probability recovers an unbiased relevance estimate.
• The correction needs propensities only for the clicked documents, not for every document in the list, which is what makes the counterfactual estimator practical at all.
• A click at rank 10 with examination probability 0.05 is weighted as 20 clicks, compensating for the 19 times in 20 an equally relevant document there went unseen.
• The price is variance: weights of 20 to 50 let a handful of deep clicks dominate the gradient, so clipping is mandatory and effective sample size, not raw log volume, is the number to report.
• Deterministic policies leave no counterfactual. A system serving argmax has no exploration, so no reweighting is possible, and propensities cannot be reconstructed after the fact.
• Position bias is not the only bias. Trust bias breaks the multiplicative factorisation outright, presentation bias moves clicks independent of relevance, and selection bias, documents never surfaced at all, cannot be reweighted because there is nothing to reweight.
• The loop has three separable effects, popularity amplification, preference narrowing and self-perpetuating uncertainty, and only the third has a clean solution.
• Airbnb reported that interleaving and counterfactual evaluation together increased experiment sensitivity by up to a factor of 100 relative to conventional A/B testing, in the first production deployment evaluating both side by side.

At a Glance

flowchart LR
    R["Ranker"] --> I["Impressions<br/>at chosen positions"]
    I --> C["Clicks"]
    C --> L["Training logs"]
    L --> R
    L --> P["Propensity model<br/>P(examined | rank)"]
    P --> W["Reweight clicks<br/>by 1 / propensity"]
    W --> R2["Debiased ranker"]
    E["Exploration slice"] -.->|"the only source of<br/>counterfactual data"| P
    E -.-> I

    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff

    class R,I,C,L slate
    class P,W purple
    class R2 emerald
    class E amber

The cycle on the left is the loop. Everything that breaks it enters from the exploration slice at the bottom, and a system without one has no entry point.

How the Field Learned to Distrust Its Own Logs

Early web search was evaluated the way classical IR had been evaluated since Cranfield: human assessors judged documents against queries, and systems were compared on those judgments. That works and it is expensive, so as soon as click logs became abundant the field tried to use them as relevance labels.

The trouble was visible almost immediately. Joachims and colleagues ran eye-tracking studies in the mid-2000s showing that users examine results top-down with sharply decaying attention, and that clicks are informative about relative preference between examined results rather than about absolute relevance. That reframing, clicks as pairwise preferences among examined items, was the first serious attempt to extract signal from biased feedback.

Click models followed: the cascade model, and later the dynamic Bayesian network model, posited an explicit generative story for browsing and inferred latent relevance from observed click patterns. They worked well on head queries with many repetitions and had nothing to say about a query seen once.

The counterfactual framing arrived a decade later and changed the shape of the problem, from "model the user well enough to invert their behaviour" to "reweight the observations by the probability they were observed".

timeline
    title From Click Logs to Counterfactual Estimation
    2005 : Joachims et al. use eye tracking to show top-down examination with steep decay
         : Clicks reframed as relative preferences among examined results
    2008 : Hu, Koren and Volinsky separate binary preference from confidence for implicit feedback
         : Cascade and DBN click models infer latent relevance from browsing patterns
    2009 : Rendle et al. introduce BPR, optimising pairwise ranking from implicit signals
    2012 : Chapelle et al. validate interleaving at scale as a high-sensitivity online comparison
    2016 : Joachims, Swaminathan and Schnabel derive unbiased LTR via propensity-weighted ERM
    2018 : Result randomisation and intervention harvesting mature as propensity estimators
    2025 : Airbnb reports interleaving and counterfactual evaluation in production side by side

The 2016 paper is the pivot (Joachims, Swaminathan and Schnabel, 2017, Unbiased Learning-to-Rank with Biased Feedback, WSDM, arXiv:1608.04468). Its contribution was to show that a propensity-weighted empirical risk minimisation is unbiased for the full-information ranking risk, and that this holds even when queries never repeat, which is exactly where click models had nothing to offer. Click models became propensity estimators rather than relevance estimators, and the theory came from causal inference rather than from user modelling.

[IMAGE: Eye-tracking heatmap over a search results page, showing dense fixation on the first two results, moderate attention on results three to five, and near-zero below the fold. Alongside it, a bar chart of click-through rate by rank on the same page, with the two curves overlaid to show they have the same shape. Caption: "Attention decays steeply. Clicks inherit that shape whether or not relevance does."]

The Examination Hypothesis and What It Buys

The standard model factorises a click into two independent events, that the user examined the position, and that the document was relevant:

Two properties make this useful. The propensity depends only on rank, not on the document, so it can be estimated once for the interface rather than per item. And because the distortion is multiplicative and known, dividing by it recovers an unbiased estimate.

The hypothesis is a simplification with known defects. Examination in reality depends on the snippet, on whether an answer appeared above, and on the user's task. Its value is that it is wrong in bounded, arguable ways and that it makes the correction tractable, which is the same trade every useful model makes.

[IMAGE: Two panels showing the same ten-result page. Left, "what the log records": clicks concentrated on ranks 1 to 3, everything below shown in grey with a single label "not clicked". Right, "what the log means": the same page with each rank annotated by its examination probability, and the grey region split into "examined, judged irrelevant" (small) and "never examined" (large). Caption: "A non-click at rank 10 is almost always a statement about attention, not about relevance."]

The estimator

Consider a ranking risk you want to minimise, say the average rank of relevant documents. With complete relevance labels it is a sum over all relevant documents. With clicks you observe only clicked ones, and each was observed with probability \(p_k\). The inverse-propensity-weighted risk

is unbiased for the full-information risk. The critical structural feature is that it requires propensities only for clicked documents. You never need to know the examination probability of a document nobody clicked, which is what makes the estimator computable from ordinary logs.

The absence of a click contributes nothing, and that is correct rather than wasteful. A non-click at rank 10 is ambiguous between "examined and judged irrelevant" and, far more often, "never examined at all". Treating absent clicks as negative labels teaches the model that deep results are bad, which is a statement about position, not quality.

[IMAGE: Line chart of examination probability against rank, from 1.00 at rank 1 decaying to about 0.05 at rank 10, plotted alongside its reciprocal on a secondary axis rising from 1 to 20. A shaded band marks where clipping at a 0.1 floor truncates the weight. Caption: "The correction and the variance problem are the same curve read two ways."]

Estimating the propensities

This is where implementations actually fail, well before the theory becomes the limitation.

Result randomisation swaps a result to a random position on a small traffic slice, so the same document is observed at multiple ranks and the click ratio identifies \(p_k\) directly. Cleanest method, and it visibly degrades results for the users in that slice.

Swap interventions are the gentler version: swap only adjacent pairs, or only within the top few positions, sacrificing much less quality for a noisier estimate.

Intervention harvesting exploits variation that already exists. Every ranker update and every A/B test places the same document at different ranks across the logs, and that natural variation identifies propensities with no deliberate degradation at all. It requires that the logs record which ranker served each impression, which many systems never implemented.

EM-based joint estimation infers propensities and relevance together from click logs alone under the examination model. It needs no intervention and its identification is weaker, resting entirely on the model being correct.

sequenceDiagram
    participant U as User
    participant S as Serving
    participant E as Exploration
    participant L as Logs
    participant T as Training
    S->>U: ranked list (argmax policy)
    U->>L: click at rank k
    Note over L: propensity unknown:<br/>deterministic policy
    E->>S: 1% slice with randomised swaps
    S->>U: perturbed list
    U->>L: click at rank k, with logged p_k
    L->>T: reweight clicks by 1 / p_k
    Note over T: unbiased ranking risk,<br/>high variance
    T->>T: clip propensities at a floor
    Note over T: bounded bias,<br/>usable variance
    T->>S: debiased ranker

By the Numbers

Sources: Joachims, Swaminathan and Schnabel, 2017, WSDM, arXiv:1608.04468; Chapelle, Joachims, Radlinski and Yue, 2012, ACM TOIS 30(1); Airbnb, 2025, KDD, arXiv:2508.00751. The "20 to 50" weight range is a characterisation of typical propensity curves rather than a measured constant, and varies substantially with interface and device mix. The Airbnb figure is a vendor-published production result and should be read as an upper bound across their metrics rather than a general expectation.

A Concrete Example

A marketplace search team wants to train a ranker on six months of logs: 40 million queries, 180 million impressions, 9 million clicks.

Step 1: the naive baseline. Train pairwise on (clicked, not-clicked) pairs from the same result page. Offline nDCG against the click labels improves 6% over the incumbent. Online A/B: flat.

The reason is visible once stated. The training labels were generated by the incumbent, so a model that reproduces the incumbent scores well on them. The 6% offline gain measures agreement with the previous ranker's choices.

Step 2: measure the propensity curve. The team runs a swap intervention on 1% of traffic for three weeks, swapping the results at ranks 1 and \(k\) for a randomly chosen \(k \in \{2,\dots,10\}\). Averaging click rates for the same document at rank 1 versus rank \(k\) gives an examination curve, normalised so \(p_1 = 1\):

Step 3: weight the clicks. Of the 9 million clicks, roughly 5.4 million are at ranks 1 to 3 with weights between 1.0 and 2.3, and about 210,000 are at ranks 8 to 10 with weights between 7.7 and 20.

Total weighted mass: the top-three clicks contribute about \(5.4\text{M} \times 1.5 \approx 8.1\) million, and the deep clicks contribute about \(210{,}000 \times 12 \approx 2.5\) million. So 2.3% of the clicks now carry roughly 24% of the training signal. That is the intended correction, and it is also the variance problem in one line.

Step 4: check the effective sample size. With this weight distribution, \(\mathrm{ESS} = (\sum w_i)^2 / \sum w_i^2\) comes out around 4.1 million against 9 million raw clicks. The estimator has the statistical power of roughly 4 million observations, not 9 million, and reporting confidence intervals from the raw count would overstate precision by about 50%.

Step 5: clip. Cap propensities at a floor of 0.1, so no weight exceeds 10. This truncates the rank-10 weight from 20 to 10, introducing a known downward bias in how much deep clicks count, and raising ESS to roughly 6.3 million. The bias is bounded and stated; the variance reduction is large. This trade is not optional in practice, and pretending the unclipped estimator is usable is the most common way the method is misapplied.

Step 6: retrain and re-evaluate. Offline nDCG against the raw click labels now drops by 2%, because the model has stopped optimising for agreement with the incumbent. Offline nDCG against propensity-weighted labels rises 9%. Online A/B on the debiased model: a 1.4% lift in a session-success metric, which replicates.

The instructive part is step 6's first sentence. The correct model looks worse on the naive offline metric, which means any team using that metric as a gate would have rejected it. The debiasing has to reach the evaluation as well as the training, or the pipeline filters out its own improvement.

[IMAGE: Two stacked bar charts side by side. Left: raw click counts by rank, dominated by ranks 1 to 3. Right: propensity-weighted training signal by rank, where ranks 8 to 10 expand to about a quarter of the total mass. Beneath both, a small annotation showing effective sample size falling from 9M to 4.1M and recovering to 6.3M after clipping. Caption: "The correction is exactly the reweighting you wanted and exactly the variance you did not."]

[IMAGE: Small-multiples chart of catalogue exposure over eight retraining cycles. Each panel is a Lorenz curve of impressions across items, bowing progressively further from the diagonal, with the Gini coefficient printed above each. A second series in a contrasting colour shows the same system with a 2% exploration slice, where the curves stay nearly stationary. Caption: "Popularity amplification is measurable with one number per retraining cycle."]

The Loop Beyond Ranking

Position bias is the search-shaped instance of a more general structure. In recommendation the same loop produces three separable effects, and conflating them makes each harder to address.

Popularity amplification. Popular items receive more impressions, more interactions, more training signal, and higher scores, compounding across retraining cycles. It is directly measurable: track the Gini coefficient of impressions across the catalogue over time, and a rising value is the loop operating. Without explicit correction, a recommender drifts toward a popularity ranking wearing a personalisation veneer.

Preference narrowing. For an individual user, an inferred interest generates more evidence for itself and less for anything else, so represented preferences narrow relative to actual ones. This is what "filter bubble" usually means, and the empirical support is genuinely mixed: establishing it requires a counterfactual about what the user would have consumed otherwise, and several studies find algorithmic recommendation narrows exposure less than self-selection does.

Self-perpetuating uncertainty. An item the model is unsure about is shown rarely, so it accumulates little evidence, so the model stays unsure. This is a bandit problem and the only one of the three with a clean solution: uncertainty-driven or randomised exploration produces exactly the evidence that is missing.

stateDiagram-v2
    [*] --> Deployed
    Deployed: Deterministic policy, no propensities
    Deployed --> Instrument
    Instrument: Add exploration and log assignment probabilities
    Instrument --> Estimate
    Estimate: Estimate the propensity or examination curve
    Estimate --> Debias
    Debias: Reweight training AND evaluation
    Debias --> Clip
    Clip: Clip weights, report effective sample size
    Clip --> Validate
    Validate: Online experiment, not offline metric
    Validate --> Deployed: propensities drift, re-estimate
    Validate --> [*]

The self-loop in that diagram matters. A debiased model reorders results, which changes user behaviour, so the propensities estimated under the old ranking no longer describe the new one. A UI change that adds an answer box above the results alters the examination curve materially. Propensity estimation is a maintained artefact, not a one-time calibration, and a stale curve corrects confidently toward the wrong target.

Where It Breaks

Trust bias breaks the factorisation outright

If users click top results partly because they are top, then a click is not examination times relevance, and dividing by examination probability over-corrects: some of what you attributed to visibility was a genuine boost from placement. Affine correction models extend the framework with an additive term, which restores validity at the cost of a second quantity to estimate from the same limited interventions.

This is not a corner case. Position conveys an editorial signal, and users respond to it.

Selection bias is untouched by any reweighting

The framework corrects for documents being examined with varying probability. Documents the candidate generator never surfaced have propensity zero, generate no observations, and cannot be reweighted, because there is nothing to reweight. Candidate generation quality bounds everything downstream, and no amount of ranking sophistication reaches past it.

This is the failure that receives the least attention relative to its size. In most production systems, retrieval recall is a larger lever than ranker quality, and it is measured far less often.

Variance is the practical limit, not the theory

The unbiasedness is real and the precision is often unusable. Weights in the tens let a handful of observations dominate, and the standard response, clipping, trades a bounded bias for a large variance reduction. Reporting the effective sample size alongside any off-policy number is the minimum discipline, and a million logged events with an ESS of 300 is a 300-sample estimate however it is presented.

Deterministic serving leaves nothing to work with

A production system serving argmax has no exploration and therefore no counterfactual. Propensities cannot be reconstructed after the fact from logs that never recorded them. The structural fix, epsilon-greedy exploration, Thompson sampling, or a small randomised traffic slice with logged probabilities, must be built before the evaluation question is asked, and it is a product cost as well as an engineering one because some users deliberately get worse results.

Exploration is also unevenly costly. Randomised recommendations are cheap on a feed with dozens of daily impressions and expensive on a surface with one high-stakes recommendation per session. A uniform exploration rate across a product is a common mis-specification.

The evaluation must be debiased too

Step 6 of the worked example is the trap that catches teams who do everything else correctly. A debiased model scores worse on naive click-based offline metrics, because those metrics reward agreement with the logging policy. If the launch gate is the naive metric, the pipeline rejects its own improvement, and the team concludes debiasing does not work.

Correction assumes the loop is the whole problem

Some concentration is genuine: popular items are often popular because they are good. Debiasing toward uniform exposure treats real quality differences as artefacts. The target is correcting the amplification the loop adds, not equalising the catalogue, and a correction strong enough to flatten exposure has stopped being a debiasing method and become a different product decision.

Alternative Designs

The Airbnb result is worth reading as evidence about how these compose rather than compete. They deployed interleaving and counterfactual evaluation together in production, evaluating both side by side and using both daily, and reported sensitivity gains of up to a factor of 100 relative to traditional A/B testing depending on approach and metric (Airbnb, 2025, KDD, arXiv:2508.00751). The architecture that emerges is a ladder: counterfactual estimation and interleaving as cheap high-sensitivity filters over many candidates, and A/B tests reserved for the few that survive, because only an A/B test measures the business quantity a launch decision needs.

[IMAGE: Funnel diagram of an evaluation ladder. Top tier, wide: "counterfactual offline estimation, dozens of candidates per week, no traffic cost". Middle tier: "interleaving, several candidates, small traffic, preference signal only". Bottom tier, narrow: "A/B test, one or two candidates, full traffic, business metrics". Each tier annotated with its sensitivity and what it cannot measure. Caption: "Three evaluation methods, three different questions, used in increasing order of cost."]

[IMAGE: Two-panel offline-versus-online comparison. Left panel: bar chart of offline nDCG against raw click labels, where the naive model scores highest and the debiased model scores 2% lower. Right panel: bar chart of an online session-success metric from the A/B test, where the ordering reverses and the debiased model wins by 1.4%. A red gate icon sits between them, labelled "a launch gate on the left panel rejects the right panel's winner". Caption: "If the evaluation is not debiased, the pipeline filters out its own improvement."]

Insights Worth Remembering

1. A ranker trained on its own logs converges toward its predecessor, not toward relevance. The offline metric improves because the metric rewards agreement with the labels, and the labels were produced by the model being replaced. Nothing in the pipeline flags this.

2. A non-click is not a negative. At rank 10 with examination probability around 0.05, the overwhelming majority of non-clicks are unexamined. Treating them as negative labels teaches the model that deep results are bad, which is a fact about position.

3. The correction needs propensities only for clicked documents. This is the structural property that makes counterfactual LTR practical rather than theoretical, and it is the reason it works where click models cannot, on queries that never repeat.

4. Effective sample size, not log volume, is the honest measure. Weights in the tens concentrate the estimate on a small fraction of observations. Reporting a confidence interval computed on the raw count is the standard way an off-policy comparison overstates itself.

5. Debiasing must reach the evaluation, not just the training. A correctly debiased model looks worse on naive click metrics. A launch gate built on those metrics rejects the improvement, and the team learns the wrong lesson from a correct result.

6. The propensity curve is a maintained artefact. A debiased ranker changes behaviour, and any UI change alters examination. Estimating the curve once and reusing it corrects confidently toward a system that no longer exists.

7. Selection bias is bigger than position bias and gets less attention. Reweighting handles documents seen with varying probability. It does nothing for documents never surfaced, and in most systems candidate-generation recall is the larger lever.

8. Exploration is the entry point, and it must predate the question. Propensities cannot be reconstructed from logs that never recorded them. A deterministic system cannot be retrofitted with counterfactual evaluation; it can only start logging and wait.

Open Questions

How much of the reported gain in ranking research survives debiasing? It is established that naive click-trained models inherit the logging policy's preferences, and established that debiased models score lower on naive metrics. What is not established is the aggregate size of the distortion across published offline ranking results, because most public benchmarks do not carry the propensity information that would allow a re-analysis.

What is the right propensity model when examination is content-dependent? The rank-only factorisation is known to be a simplification, and affine models patch trust bias specifically. Whether a richer propensity that conditions on snippet features and on what appeared above can be estimated reliably from realistic intervention budgets is an open empirical question, and the gap between "the model is wrong" and "the correction is worse than nothing" has not been well characterised.

Can the exploration budget be optimised rather than chosen? Exploration costs short-term quality and buys future learning, which is a bandit problem one level up. Some work treats it this way; whether the optimal allocation is stable enough to be worth computing, against a simple fixed slice, is not settled in production settings.

Do these corrections transfer to retrieval inside LLM systems? A RAG pipeline's retriever is selected by relevance and its output is consumed by a generator, with user feedback arriving on the generated answer rather than on the retrieved documents. Attribution from an answer-level signal back to individual retrieved passages is not the same problem as position bias, and the counterfactual machinery has not been systematically adapted to it.

How should the three loop effects be traded against each other? Popularity amplification, preference narrowing and self-perpetuating uncertainty have different corrections with different costs, and interventions for one can worsen another. Whether there is a principled joint objective, as opposed to a set of separately tuned penalties, is an open design question rather than a solved one.

Sources and Further Reading

1. Joachims, T., Swaminathan, A., & Schnabel, T. (2017). "Unbiased Learning-to-Rank with Biased Feedback." WSDM '17. arXiv:1608.04468
2. Chapelle, O., Joachims, T., Radlinski, F., & Yue, Y. (2012). "Large-Scale Validation and Analysis of Interleaved Search Evaluation." ACM TOIS, 30(1). DOI
3. Airbnb (2025). "Harnessing the Power of Interleaving and Counterfactual Evaluation for Airbnb Search Ranking." KDD '25. arXiv:2508.00751
4. Hu, Y., Koren, Y., & Volinsky, C. (2008). "Collaborative Filtering for Implicit Feedback Datasets." ICDM '08. DOI
5. Rendle, S., Freudenthaler, C., Gantner, Z., & Schmidt-Thieme, L. (2009). "BPR: Bayesian Personalized Ranking from Implicit Feedback." UAI '09. arXiv:1205.2618
6. Koren, Y., Bell, R., & Volinsky, C. (2009). "Matrix Factorization Techniques for Recommender Systems." IEEE Computer, 42(8), 30-37. DOI
7. Burges, C. J. C. (2010). "From RankNet to LambdaRank to LambdaMART: An Overview." Microsoft Research Technical Report MSR-TR-2010-82. Microsoft Research
8. Järvelin, K., & Kekäläinen, J. (2002). "Cumulated gain-based evaluation of IR techniques." ACM TOIS, 20(4), 422-446. DOI
9. Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., & Gurevych, I. (2021). "BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models." arXiv:2104.08663
10. Chernozhukov, V., Chetverikov, D., Demirer, M., et al. (2018). "Double/debiased machine learning for treatment and structural parameters." The Econometrics Journal, 21(1), C1-C68. DOI
11. Rosenbaum, P. R., & Rubin, D. B. (1983). "The Central Role of the Propensity Score in Observational Studies for Causal Effects." Biometrika, 70(1), 41-55. DOI
12. Robertson, S., & Zaragoza, H. (2009). "The Probabilistic Relevance Framework: BM25 and Beyond." Foundations and Trends in Information Retrieval, 3(4), 333-389. DOI

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.