Why Offline Gains Vanish Online

A reranker improves nDCG@10 by 4% on the held-out judgments. It ships to an experiment and moves nothing measurable. This is common enough in search and recommendation to be the expected outcome rather than an anomaly, and treating it as a mystery wastes the diagnostic information available. There are four distinct mechanisms, and they call for different responses.

The metric does not map to the outcome

nDCG measures ranking quality against judgments; the business metric measures behaviour. The relationship between them is monotone at best and often flat over the range where a system operates. Moving nDCG from 0.71 to 0.74 may be entirely within the region where users find what they need either way, so a real ranking improvement produces no behavioural change.

The diagnostic is to establish the sensitivity curve empirically: from past experiments, plot measured offline deltas against measured online deltas. Most teams that do this discover that offline improvements below some threshold have never once produced a detectable online effect, which converts a mysterious failure into a launch criterion.

The judgments do not represent your users

Judgments are made by assessors, or derived from behaviour under a previous system, or borrowed from a public collection. Any of those may encode a notion of relevance that differs from what users of this product at this moment actually want. Freshness, locality, personalisation and task context are routinely under-weighted in judgments and heavily weighted by users.

The change is real, and something else absorbs it

Two versions of this. Position effects dominate small reordering: swapping ranks 3 and 4 barely changes what users see, and if the previous rank 3 was adequate the user is satisfied either way. The system compensates: a query that returns a mediocre first result is followed by a reformulation, so the session succeeds regardless and session-level metrics do not move even though the first result improved.

The offline evaluation was optimistic

The unglamorous and frequent cause. Judgment pools were built from systems resembling the baseline, so the new system's novel finds counted as irrelevant, or, in the other direction, hyperparameters were selected on the same data used for the final number. Sparse labels make the second especially easy: with one relevant passage per query, repeated evaluation on the same set is a slow leak.

When it breaks

The inverse failure is more dangerous. A change that hurts offline metrics and wins online is real and frequently rejected. Freshness boosts, diversity injection and personalisation all degrade nDCG against static judgments while improving the user experience, because the judgments have no notion of when or for whom.

"No measurable effect" is often "insufficient power". A ranking change producing a 0.2% shift in click-through requires substantial traffic to detect, and most search experiments are not powered for effects that size. Concluding the change did nothing, when the experiment could not have detected what was expected, is a different error from the four above and needs a power calculation, not a diagnosis.

Offline metrics still earn their place. They are cheap, fast, and reproducible, and they catch regressions before traffic is spent. The correct posture is to treat them as a filter with a known and measured relationship to online outcomes, not as a proxy that has been assumed to work and never checked.

The relationship drifts. A sensitivity curve estimated two years ago described a different index, interface and population. Like propensities, it is a maintained artefact rather than a constant of the system.