Offline Evaluation and Sampled Metrics in Recommenders

Recommender A places the relevant item at rank 100 out of 10,000 for every test user. Its exact Recall@10 is zero. Evaluate it the popular way, ranking the relevant item against 99 randomly sampled items, and its Recall@10 becomes 1.000. Recommender C, which has an exact Recall@10 of 0.200, scores only 0.567 under the same protocol. The sampled metric has not added noise; it has reversed the conclusion (Krichene & Rendle, 2020, On Sampled Metrics for Item Recommendation, KDD, doi:10.1145/3394486.3403226).

Offline evaluation is the only affordable way to compare dozens of models, and it is where a large share of the field's claimed progress has come from. It fails in three distinct ways: the metric, the split, and the baselines.

Why sampled metrics lie

Ranking every item for every test user costs a forward pass per item, so papers often rank the held-out item against \(m\) random negatives instead. Let \(n\) be the catalogue size and \(r\) the item's true rank. A uniformly sampled irrelevant item ranks above it with probability

so the number of sampled items above it is \(\text{Binomial}(m, p)\), and the sampled rank is \(\tilde{r} = 1 + \text{Binomial}(m, p)\).

Replay recommender A. With \(r = 100\), \(n = 10{,}000\) and \(m = 99\), \(p = 99/9999 \approx 0.0099\) and the expected number of sampled items above is about 0.98. The relevant item almost always lands in the sampled top 10, so a model that never puts the item in the real top 10 scores perfectly.

The general result is worse than a bias. Krichene and Rendle show that sampled versions of Recall, NDCG and average precision are inconsistent: the relative order of two recommenders is not preserved even in expectation, and the ordering can change as \(m\) changes. As \(m\) shrinks, every metric collapses toward AUC, which weights all rank positions linearly and ignores the top-heaviness that made NDCG worth using. AUC is the only one of these metrics whose expected value survives sampling. The authors propose bias-corrected estimators, and their recommendation is to avoid sampling where possible.

This is a live disagreement in practice. Sampled evaluation remains common because it is cheap and because some argue that rankings are usually stable on real models. The theory says stability is an empirical accident of particular models, not a property of the protocol.

How the split leaks the future

The second failure is temporal. Random or per-user leave-one-out splits hold out each user's last interaction, but they do not align users on a global timeline. User A's test item may be from 2015 while user B's training data runs to 2018, so the model learns from interactions that happened after the moment it is supposedly predicting.

Ji and colleagues tested BPR, NeuMF, SASRec and LightGCN on four popular datasets and found that all of them recommend items that did not yet exist at prediction time, and that this leakage changes measured accuracy unpredictably, reordering models in some cases (Ji et al., 2023, A Critical Study on Data Leakage in Recommender System Offline Evaluation, ACM TOIS 41(3), arXiv:2010.11060). Their proposed fix is a global timeline split: train on everything before a cutoff, test after it. That matches deployment and is noticeably harder, which is why leaked numbers look better.

Baselines that nobody tuned

Dacrema, Cremonesi and Jannach tried to reproduce 18 neural recommendation algorithms from top venues. Only 7 could be reproduced with reasonable effort, and 6 of those 7 were often beaten by simple nearest-neighbour or graph heuristics; the remaining one did not consistently beat a well-tuned linear method (Dacrema et al., 2019, Are We Really Making Much Progress?, RecSys, arXiv:1907.06902). Rendle and colleagues found the same pattern for neural collaborative filtering: a carefully tuned dot-product model substantially outperformed the learned MLP similarity it was claimed to supersede (Rendle et al., 2020, Neural Collaborative Filtering vs. Matrix Factorization Revisited, arXiv:2005.09683).

The mechanism is mundane. Authors tune their own model for weeks and run the baseline at defaults from another paper, often with a different split. None of this requires bad faith. It requires only that the tuning budget is not reported.

When it breaks

Offline gains do not predict online gains. Held-out interactions were produced by the recommender that was live when the data were logged, so offline accuracy rewards imitating it. This is the evaluation face of the loop described in feedback loops and filter bubbles, and it is why A/B tests remain the arbiter.

Preprocessing changes the problem. Filtering to users and items with at least \(k\) interactions removes exactly the cold and sparse cases that production faces, and papers vary \(k\) without comment.

The metric must match the surface. NDCG@10 suits a ranked list; a carousel of five tiles or a single autoplay slot rewards something else. Choosing a metric is choosing which errors matter.

Full ranking has its own cost. Scoring a million-item catalogue for every test user is expensive, which is the honest reason sampling persists. Exact evaluation on a random subset of users is unbiased where sampling items is not, and it is the cheaper correct option.