The Progress Illusion in Recommender Systems: Weak Baselines, Sampled Metrics and Leaky Splits

In April 2012 Netflix explained what it had done with the million-dollar answer it paid for. The Netflix Prize asked for a 10% cut in the RMSE of its Cinematch predictor, from 0.9525 to 0.8572. The winning ensemble blended hundreds of models, and Netflix did not ship it: "the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment" (Amatriain & Basilico, 2012, Netflix Recommendations: Beyond the 5 stars (Part 1), Netflix TechBlog). What went into production was two components from the 2007 Progress Prize: a matrix factorisation at 0.8914 and a restricted Boltzmann machine at 0.8990, blended to 0.88.

That was the benign version of the gap between an offline number and a real improvement. In the decade that followed, the gap got worse for reasons unrelated to engineering cost: reported gains began to come from the measurement itself.

Why this matters: Anyone who picks an architecture from the literature, or promotes a model on an offline dashboard, is trusting a protocol. Three protocol choices (how hard the baseline was tuned, how many items the metric ranks against, whether the split respects time) can each produce a gain larger than the real difference between two reasonable models.

TL;DR

• Most neural top-N claims that were re-examined did not survive tuned baselines. Of 26 neural methods from top venues, 12 were reproducible, and 11 of those lost to simple nearest-neighbour or linear methods.
• Baselines lose because they are under-tuned. A carefully run Bayesian matrix factorisation scored 0.7633 RMSE on MovieLens 10M, beating the best result ever reported there (0.7634).
• Sampled metrics are biased in a model-dependent way. Ranking the target against 100 random negatives turns HR@10 into something close to AUC and can reverse two models' order in expectation.
• The reversal happens on real models. One BERT4Rec implementation was 33% worse than MF-BPR on sampled Recall@10 on Steam and 86% better on full Recall@10.
• Common splits leak the future. Of 85 papers, 53.7% used random splits and 28% leave-one-out; 12.3% split by time. Under leakage, metric shifts reached 89.5%.
• Full ranking is cheap. 887,369 recommendation lists over 172,756 items took 205.1 seconds on one GPU.

At a Glance

flowchart LR
    T["True model difference"]:::blue
    B["Baseline tuning gap"]:::amber
    S["Sampled-metric bias"]:::amber
    L["Temporal leakage"]:::amber
    R["Reported gain"]:::purple
    O["Online A/B outcome"]:::teal
    F["Tune, rank all, split by time"]:::emerald
    T --> R
    B --> R
    S --> R
    L --> R
    R -.->|"weak predictor"| O
    F --> T
    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff

A reported improvement is a sum. Only the blue term is about the model; the amber terms are properties of the experiment, and for a decade they were routinely as large.

[IMAGE: A stacked horizontal bar labelled "Reported gain +8%" split into four segments: a small blue "true difference" and three amber segments "untuned baseline", "sampled metric", "leaked future". Caption: "Only one term in the results table is about the model."]

From RMSE Leaderboards to the Reproducibility Crisis

The Netflix Prize's durable lesson was calibration. Many teams attacking one split with a hidden test set slowly learned what a well-run baseline scores. Rendle, Zhang and Koren reconstructed that curve: early SVD results near 0.94 probe RMSE, Simon Funk's regularised SGD factorisation at 0.93, and eventually plain matrix factorisation at 0.8985 with ALS (Rendle, Zhang & Koren, 2019, On the Difficulty of Evaluating Baselines, arXiv:1905.01395). The same method spanned four hundredths of RMSE depending on who ran it, while the progress prizes moved only from 0.8712 in 2007 to 0.8616 in 2008. The split also encoded time: each user's last six ratings were withheld, making evaluation a forecasting problem.

Then research moved to implicit feedback and ranking metrics, which are expensive to compute exactly because every test case scores the whole catalogue. Neural Collaborative Filtering adopted a cheaper protocol in 2017: hold out each user's last interaction and rank it against 100 sampled items (He et al., 2017, Neural Collaborative Filtering, WWW). The protocol spread with the model, and the shared benchmark dissolved. Ferrari Dacrema, Cremonesi and Jannach counted over 20 datasets in the papers they examined, most used once or twice, with protocols chosen, in their words, "not well justified beyond the fact that others used them before" (Ferrari Dacrema, Cremonesi & Jannach, 2019, Are We Really Making Much Progress?, RecSys, arXiv:1907.06902). They could reproduce 7 of 18 neural methods, and 6 of the 7 were often beaten by simple heuristics.

timeline
    title How recommender evaluation lost and regained calibration
    2006 : Netflix Prize opens with a shared split and hidden test set
         : McNee, Riedl and Konstan argue accuracy is not enough
    2007 : First Progress Prize from a blend of 107 algorithms
    2012 : Netflix explains why the Grand Prize blend was never deployed
    2014 : swissinfo.ch finds most-popular best offline and worst online
    2017 : NCF popularises leave-one-out with 100 sampled negatives
    2019 : Only 7 of 18 neural recommenders reproducible
         : Tuned matrix factorisation beats five years of MovieLens 10M results
    2020 : Sampled metrics shown inconsistent even in expectation
         : Tuned dot product beats learned MLP similarity
    2022 : BERT4Rec review of 40 papers and 134 comparisons
    2023 : Leakage study in TOIS and four widespread flaws catalogued

[IMAGE: Line chart of Netflix Prize RMSE over time: Cinematch at 0.9525, Progress Prizes at 0.8712 (2007) and 0.8616 (2008), and the 0.8572 target as a dashed line. Caption: "A shared split made a 0.01 RMSE gain a year of work."]

How the Illusion Is Manufactured

Each channel has its own mathematics. None requires bad faith, only an unexamined default.

Channel one: the tuning budget asymmetry

A proposed model is tuned by its authors for weeks; baselines often run with hyperparameters copied from a paper with a different dataset. Suppose each configuration's validation score is the model's true quality \(\mu\) plus noise of standard deviation \(\sigma\) from seeds, splits and hyperparameter luck. The best of \(K\) tries then has expectation approximately

This is the Gaussian extreme-value approximation, an estimate rather than an exact result, but its shape is what matters. A model searched over \(K = 50\) configurations gains about \(2.8\sigma\) over its own mean; a baseline run once gains nothing. If the true gap is smaller, the table shows a win for whichever side was searched harder.

The record matches. Rendle, Zhang and Koren re-ran vanilla SGD matrix factorisation on MovieLens 10M and got 0.7720, against 0.803 reported for the same model family and better than newer methods such as LLORMA (0.7815) and AutoRec (0.782). Bayesian matrix factorisation, previously reported at 0.8197, reached 0.7633. For scale, the community took about four years to move that benchmark from 0.7815 to 0.7634. The extended study by Ferrari Dacrema and colleagues found 11 of 12 reproducible neural approaches outperformed by simple methods, and none consistently better than matrix factorisation or linear models (Ferrari Dacrema et al., 2021, A Troubling Analysis of Reproducibility and Progress, ACM TOIS, arXiv:1911.07698). Doing it properly took 41,000 model fits, about 253 days of compute. That cost is the honest reason baselines go untuned.

NCF itself became the emblem. Rendle, Krichene, Zhang and Anderson found that "with a proper hyperparameter selection, a simple dot product substantially outperforms the proposed learned similarities", and that an MLP needs large data and width to approximate a dot product at all (Rendle et al., 2020, Neural Collaborative Filtering vs. Matrix Factorization Revisited, RecSys, arXiv:2005.09683). Because NCF had become a standard baseline, every model compared against it inherited its weakness as a margin.

Channel two: the sampled metric is a different metric

Let the catalogue hold \(n\) items and let a model place the relevant item at true rank \(r\). Exact HR@\(k\) is \(\mathbb{1}[r \le k]\). The sampled protocol draws \(m\) irrelevant items uniformly and ranks the target among \(m + 1\). Each sampled item outranks the target with probability

so the sampled rank is \(\tilde r = 1 + X\) with \(X \sim \text{Binomial}(m, p(r))\) (Krichene & Rendle, 2020, On Sampled Metrics for Item Recommendation, KDD). The expected sampled hit rate at true rank \(r\) is

This is a smooth S-curve in \(r\) rather than a step at \(k\), with its midpoint where \(mp \approx k\), that is at \(r \approx k(n-1)/m\). With \(n = 10{,}000\), \(m = 100\) and \(k = 10\), the cutoff at rank 10 becomes a soft threshold near rank 1,000. Sampled "HR@10" asks whether the item is in the top tenth of the catalogue.

Push \(m\) to one and the metric's identity is explicit:

which is exactly the per-case AUC. Krichene and Rendle prove the general version: as the sample shrinks, recall, precision, average precision and NDCG all collapse toward AUC, a metric that values moving an item from rank 101 to 100 as much as from 2 to 1. They call a metric consistent under sampling if the order of any two recommenders on the exact metric equals their order on the expected sampled metric, and show that only AUC qualifies. The bias depends on each model's whole rank distribution, so it differs between models, and models swap order.

That is why sampled metrics produce false discoveries rather than noise. The uncorrected metrics "have low standard deviation, so the issue is not that of variance, but is due to the bias"; in their MovieLens experiment, "the worst recommender would be found to be the best one" (Krichene & Rendle, 2021, IJCAI extended abstract).

A correction exists. Since \((\tilde r - 1)/m\) estimates \(p(r)\) without bias, the rank can be estimated as

and the metric applied to \(\hat r\). This improves ordering accuracy at the cost of variance, but it cannot restore resolution the sample never had, as the worked example shows.

[IMAGE: Log-scaled x-axis of true rank from 1 to 10,000. Curve 1: exact HR@10, a hard step at rank 10. Curve 2: expected sampled HR@10 with 100 negatives, near 1 until rank 400 and falling through 0.5 near rank 1,000. The region between rank 10 and 1,000 shaded amber, labelled "hit when sampled, miss in reality". Caption: "Sampling moves the cutoff from rank 10 to roughly rank 1,000."]

Channel three: the split lets the model read the future

A deployed model trained at time \(t\) predicts behaviour after \(t\) from interactions before it. Leave-one-out holds out each user's last interaction, but users stop at different times: user A's test item may date from 2015 while user B's training history runs to 2019, and collaborative filtering transfers B's future co-occurrences onto A's past.

Ji, Sun, Zhang and Li trained BPR, NeuMF, SASRec and LightGCN on ten-year slices of MovieLens-25M, Yelp and two Amazon datasets, adding progressively more future years to training while fixing the test year (Ji et al., 2023, A Critical Study on Data Leakage in Recommender System Offline Evaluation, ACM TOIS, arXiv:2010.11060). All four models recommended items released after the test interaction. Accuracy changes had no consistent sign, reaching 89.5% for NeuMF on Amazon-music, and the models' rank order shifted. They cite the survey by Sun and colleagues of 85 papers, of which 53.7% used random splits and only 12.3% split by timepoint (Sun et al., 2020, Are We Evaluating Rigorously?, RecSys).

Leakage does not inflate every model equally, because it rewards co-occurrence memorisation. Hidasi and Czapp measured the share of test item-to-item transitions already seen in training on Rees46: 0.5254 under leave-one-out against 0.4222 under a time split (Hidasi & Czapp, 2023, Widespread Flaws in Offline Evaluation of Recommender Systems, RecSys, arXiv:2307.14951). The task gets easier by different amounts for memorising and generalising models, which is exactly the comparison the benchmark was meant to settle.

Seeing It in Motion

The three channels enter at different points of an ordinary pipeline. Each amber node is a decision usually inherited from a previous paper.

flowchart TB
    D["Raw interaction log"]:::blue
    P["k-core filtering"]:::amber
    SP["Train and test split"]:::amber
    TR["Train candidate model"]:::purple
    BL["Train baselines"]:::amber
    C["Candidate set per test case"]:::amber
    M["HR and NDCG at k"]:::purple
    ST["Seeds and significance"]:::slate
    CL["Claim of improvement"]:::teal
    D --> P --> SP
    SP --> TR
    SP --> BL
    TR --> C
    BL --> C
    C --> M --> ST --> CL
    SP -.->|"leakage"| M
    BL -.->|"tuning gap"| CL
    C -.->|"AUC-like bias"| M
    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

Followed for one test user, the sampled protocol shows where information disappears: the model never competes against the hundreds of unsampled items it would have ranked above the target.

sequenceDiagram
    participant E as Evaluator
    participant S as Negative sampler
    participant R as Recommender
    participant K as Metric
    E->>S: request 100 negatives for user u
    S-->>E: 100 random items of 10,000
    E->>R: score target plus negatives
    R-->>E: 101 scores
    Note over E,R: true rank 800, yet only about 8 sampled items outrank it
    E->>K: sampled rank 9 of 101
    K-->>E: HR@10 equals 1
    Note over K: full-ranking HR@10 for this user is 0

A claim also has a life cycle, and the worst habit sits in one transition: an unreplicated result becomes the next paper's baseline.

stateDiagram-v2
    [*] --> Published: sampled metric and default baselines
    Published --> Baseline: re-run at defaults by others
    Baseline --> Published: new model beats it
    Baseline --> Replicated: independent full-rank study
    Replicated --> Confirmed: holds against tuned baselines
    Replicated --> Overturned: tuned simple model wins
    Confirmed --> [*]
    Overturned --> [*]

By the Numbers

Sources: Ferrari Dacrema et al., 2021; Rendle, Zhang & Koren, 2019; Rendle et al., 2020; Petrov & Macdonald, 2022; Hidasi & Czapp, 2023; Ji et al., 2023; Garcin et al., 2014. The NCF-revisited HR@10 values use the sampled protocol for comparability with the original paper, so they show a tuning effect, not a full-ranking result.

The rows fall into three clusters: tuning, metric, and split. The most useful row for a practitioner is the cost one. Sampling was adopted to save compute that, on public datasets, costs minutes.

A Concrete Example

Two models are evaluated on the same 10,000-item catalogue. Model A is sharp: when it knows the next item it ranks it very high, and otherwise it has no idea. Model B is smooth and popularity-leaning: it rarely nails the item but never buries it.

Step 1: exact metrics. Full-ranking HR@10 counts ranks at or below 10: A scores \(0.25\), B scores \(0.15\). NDCG@10 is \(1/\log_2 4 = 0.5\) at rank 3, so A gets \(0.125\) and B \(0.075\). A is better at what a ten-slot product needs. Per-case AUC, \((n - r)/(n - 1)\), says the opposite: A \(0.837\), B \(0.973\), because B never buries anything.

Step 2: per-rank probabilities with \(m = 100\) uniform negatives.

Replay the rank-800 row: \(X \sim \text{Binomial}(100, 0.0799)\) has mean 7.99, and the chance that nine or fewer random items beat the target is 0.723. A clean miss counts as a hit nearly three times in four.

Step 3: sampled HR@10.

The true order, A at 0.25 over B at 0.15, has reversed, and the sampled numbers now track the AUC ordering from step 1. These are expectations, so a thousand seeds converge on the wrong answer.

Step 4: apply the rank correction. With \(\hat r = 1 + 9{,}999(\tilde r - 1)/100\), a corrected hit at \(k = 10\) requires \(\tilde r = 1\), so corrected HR@10 is the \(P(X = 0)\) column:

The gap shrinks from 0.58 to 0.015, but the order is still wrong. One sampled position stands for about \(n/m = 100\) true positions, so B's mass at rank 120 lands in the "rank 1" bucket 30% of the time. A question about the top 10 of 10,000 is below the protocol's resolution.

Step 5: the cost of the right answer. Scoring all 10,000 items for 6,000 test users is 60 million dot products for a factorisation model, a trivial GPU workload.

[IMAGE: Two rank histograms on a shared log axis, model A in purple with spikes at 3, 40 and 2,500, model B in teal with spikes at 3, 120 and 800. A green line at rank 10 labelled "real HR@10 cutoff" and an amber band from 700 to 1,300 labelled "effective cutoff with 100 negatives". Caption: "Where each model's mass sits relative to the two cutoffs decides the winner."]

Where It Breaks

Replications inherit the same hazards

Petrov and Macdonald reviewed 40 papers with 134 BERT4Rec-versus-SASRec comparisons: BERT4Rec won 64%, SASRec 24%, the rest tied (Petrov & Macdonald, 2022, A Systematic Review and Replicability Study of BERT4Rec, RecSys, arXiv:2207.07483). Much of the disagreement traced to implementations: the original code needed up to 30 times its default training time to match published numbers, and on MovieLens-1M one reimplementation trained in 18 minutes and scored 2.3 times higher on unsampled Recall@10 than another that trained for 5.6 hours. A replication that runs an under-trained third-party implementation is the tuning channel with roles reversed.

Datasets without the structure being tested

MovieLens 10M has day-level timestamp collisions and ships sorted by user and item ID, so colliding events are re-ordered by ID, creating identical artificial sequences for users who rated the same items in different orders. Hidasi and Czapp found that removing GRU4Rec's sequence modelling cut Recall@5 by 26.18% there, while on Amazon Beauty it changed almost nothing. The sequence model was learning the preprocessing.

Preprocessing that selects the winner

In the same study, GRU4Rec beat neighbour-based V-SkNN by 5.7% in Recall@5 on full history and lost by 29.6% when both trained on the last 14 days. k-core filtering removes exactly the cold and sparse cases production faces, and papers vary \(k\) without comment.

Statistical power

On 6,000 test users, the standard error of an HR@10 near 0.2 is \(\sqrt{0.2 \times 0.8/6000} \approx 0.005\). A paper comparing eight baselines on five datasets and four metrics runs 160 comparisons, enough for several nominal wins at \(p < 0.05\) by chance.

A correct offline protocol still is not the product

At swissinfo.ch, recommending the most popular stories was the best strategy offline and the poorest online, while context-tree recommenders raised click-through by up to 35% (Garcin et al., 2014, Offline and Online Evaluation of News Recommender Systems at swissinfo.ch, RecSys). In simulation across eleven recommenders and six environments, offline metrics correlated with online performance but with diminishing returns, and rankings shifted with the amount of initial data (Krauth et al., 2020, Do Offline Metrics Predict Online Performance in Recommender Systems?, arXiv:2011.07931). A clean offline protocol is necessary for trusting a comparison, not sufficient for trusting a launch.

Accuracy is one axis of several

McNee, Riedl and Konstan warned in 2006 that "the recommendations that are most accurate according to the standard metrics are sometimes not the recommendations that are most useful to users" (McNee et al., 2006, Being Accurate Is Not Enough, CHI EA). Diversity, novelty, serendipity and coverage trade off against accuracy in ways one leaderboard number hides (Kaminskas & Bridge, 2016, ACM TiiS). In the worked example, model B's smoothness is a coverage-like property the sampled metric rewarded without anyone asking about coverage.

[IMAGE: Scatter plot, each point a published model; x-axis "reported gain over baseline", y-axis "gain after re-tuning baseline and full ranking". Most points fall below the diagonal, many below zero. Caption: "A positive x often becomes a zero or negative y."]

Alternative Designs

One row deserves a fair hearing because it marks a real disagreement. Cañamares and Castells compared target-set sizes against unbiased evaluation and found that "minimum target sets incur in substantial distortion in pairwise system comparisons, while maximum sets may not be ideal either, and better options may lie in between the extremes" (Cañamares & Castells, 2020, On Target Item Sampling in Offline Recommender System Evaluation, RecSys). The "irrelevant" items in a full ranking are really unobserved ones, and observations are missing not at random, so full ranking carries its own popularity bias. Krichene and Rendle conclude sampling should be avoided; Cañamares and Castells treat the amount as an empirical question. Both reject the 100-random-negatives default.

How It Is Used in Practice

In production the offline stage is a filter, not a verdict. Netflix in 2012 tied its recommender's value to engagement, reporting that 75% of what members watched came from recommendations, not to RMSE (Amatriain & Basilico, 2012). Jannach and Jugovac's review of field tests finds business value measured through click-through, conversion, sales and engagement, with a weak link to academic offline gains (Jannach & Jugovac, 2019, Measuring the Business Value of Recommender Systems, ACM TMIS).

A disciplined harness follows from the three channels. Split on a global timestamp and retrain on train plus validation before testing. Rank the whole catalogue, or subsample users rather than items, or score the candidate set the real retrieval stage produces. Give every model the same tuning budget and report it, with a tuned matrix factorisation, item-kNN and popularity as fixed references. Report seeds, confidence intervals and multiplicity. Track coverage and popularity concentration beside HR and NDCG. Finally, regress A/B deltas on offline deltas for past model pairs; that slope is the calibration of the harness.

[IMAGE: A checklist card titled "Evaluation harness contract" with six rows (split, candidate set, tuning budget, metrics, seeds, online calibration), each paired with the recommended default. Caption: "Write the protocol down before training anything."]

Insights Worth Remembering

1. A reported gain is a sum, and most terms are about the experiment. Ask which terms were controlled before reading the number.
2. Sampled HR@10 is not a noisy HR@10; it is a different metric. With 100 negatives among 10,000 items it asks about roughly the top 1,000, which is closer to AUC than to any carousel.
3. Bias survives repetition. Low-variance sampled metrics point consistently at the wrong winner; more seeds make a biased protocol more confident, not more correct.
4. Rank-distribution shape decides whom sampling favours. Sharp, bimodal models lose; smooth models that never bury items win. The protocol quietly selects a model family.
5. The cheapest improvement in recommender research is tuning the baseline. It is skipped because doing it properly costs tens of thousands of fits.
6. Leakage with no reliable sign is worse than inflation. A consistent bias could be discounted; shifts of up to 89.5% in either direction cannot.
7. The excuse for sampling has expired. On public benchmarks, full ranking costs minutes.

Open Questions

How much of the sequential-recommendation literature survives full ranking and time splits? Individual studies show orderings changing; no one has re-run the post-2018 literature under one harness, so the surviving fraction is unknown.

What is the right target set under missing-not-at-random data? Cañamares and Castells showed neither extreme agrees best with unbiased evaluation on their data. Whether an intermediate optimum generalises is open.

Can offline-online calibration be published? Companies hold paired offline and A/B deltas; the correlation is the most useful number in the field and is almost never released. Simulation suggests diminishing returns; real deployments are speculation until published.

Do LLM-based recommenders inherit the same protocols? Early work often reuses sampled candidates and leave-one-out splits. The same reversals are likely but not yet measured, with the added risk that pretrained models have seen public datasets' item text.

Sources and Further Reading

1. Amatriain, X., & Basilico, J. (2012). "Netflix Recommendations: Beyond the 5 stars (Part 1)." Netflix TechBlog. Link
2. Rendle, S., Zhang, L., & Koren, Y. (2019). "On the Difficulty of Evaluating Baselines." arXiv:1905.01395
3. Ferrari Dacrema, M., Cremonesi, P., & Jannach, D. (2019). "Are We Really Making Much Progress?" RecSys 2019. arXiv:1907.06902
4. Ferrari Dacrema, M., Boglio, S., Cremonesi, P., & Jannach, D. (2021). "A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research." ACM TOIS 39(2). arXiv:1911.07698
5. He, X., et al. (2017). "Neural Collaborative Filtering." WWW 2017. doi:10.1145/3038912.3052569
6. Rendle, S., Krichene, W., Zhang, L., & Anderson, J. (2020). "Neural Collaborative Filtering vs. Matrix Factorization Revisited." RecSys 2020. arXiv:2005.09683
7. Krichene, W., & Rendle, S. (2020). "On Sampled Metrics for Item Recommendation." KDD 2020. doi:10.1145/3394486.3403226; extended abstract, IJCAI 2021. PDF
8. Cañamares, R., & Castells, P. (2020). "On Target Item Sampling in Offline Recommender System Evaluation." RecSys 2020. doi:10.1145/3383313.3412259
9. Ji, Y., Sun, A., Zhang, J., & Li, C. (2023). "A Critical Study on Data Leakage in Recommender System Offline Evaluation." ACM TOIS 41(3). arXiv:2010.11060
10. Sun, Z., et al. (2020). "Are We Evaluating Rigorously?" RecSys 2020. doi:10.1145/3383313.3412489
11. Petrov, A., & Macdonald, C. (2022). "A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation." RecSys 2022. arXiv:2207.07483
12. Hidasi, B., & Czapp, Á. T. (2023). "Widespread Flaws in Offline Evaluation of Recommender Systems." RecSys 2023. arXiv:2307.14951
13. Garcin, F., et al. (2014). "Offline and Online Evaluation of News Recommender Systems at swissinfo.ch." RecSys 2014. doi:10.1145/2645710.2645745
14. Krauth, K., et al. (2020). "Do Offline Metrics Predict Online Performance in Recommender Systems?" arXiv:2011.07931
15. McNee, S. M., Riedl, J., & Konstan, J. A. (2006). "Being Accurate Is Not Enough." CHI 2006 Extended Abstracts. doi:10.1145/1125451.1125659
16. Kaminskas, M., & Bridge, D. (2016). "Diversity, Serendipity, Novelty, and Coverage." ACM TiiS 7(1). doi:10.1145/2926720
17. Jannach, D., & Jugovac, M. (2019). "Measuring the Business Value of Recommender Systems." ACM TMIS. doi:10.1145/3370082

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.