LTR Datasets and Ranking Feature Design

When Yahoo! ran its Learning to Rank Challenge in 2010, every winning team used decision trees and ensembles, and the accuracy differences between the winners were very small. The contestants never learned what they were ranking. The queries, URLs and feature descriptions were withheld, because, in the organisers' words, feature engineering is a critical component of any learning-to-rank system and search companies rarely disclose their features (Chapelle & Chang, 2011, Yahoo! Learning to Rank Challenge Overview, JMLR W&CP 14). That tension, public benchmarks built from private signals, shapes everything the standard datasets can teach.

What a benchmark row is

A learning-to-rank dataset is a set of query groups. Each row is one query-document pair: a graded relevance label, a query identifier, and a numeric feature vector. Models learn to order the rows within each query, and are evaluated with nDCG or ERR per query, averaged. Nothing in the row is text.

The major collections differ mainly in scale and in how much they disclose:

• LETOR 3.0 and 4.0 came from TREC collections: the .gov set with 575 queries and 64 features, OHSUMED with 45 features, and LETOR 4.0 with 2,476 queries from the Million Query tracks over the roughly 25-million-page Gov2 crawl with 46 features (Qin, Liu, Xu & Li, 2010, LETOR, Information Retrieval 13(4); Qin & Liu, 2013, arXiv:1306.2597).
• MSLR-WEB30K has over 30,000 queries (31,531 with about 3.77 million documents by the Yahoo overview's count), 136 documented features and labels from 0 to 4, split into five folds with three parts for training, one for validation and one for testing (Microsoft Research, MSLR).
• Yahoo! set 1 has 19,944 training, 2,994 validation and 6,983 test queries with 473,134 training documents and 519 undisclosed features; set 2 is a smaller transfer-learning set with 596 features.
• Istella LETOR has 33,018 queries, 220 undisclosed features and 10,454,629 query-document pairs labelled 0 to 4 (Dato et al., 2016, TOIS 35(2)).

The Yahoo overview records why scale mattered: researchers had noticed that conclusions from the small LETOR sets could differ from those on real data, citing a method that gained 15.8 percent nDCG@10 on TREC data and gave results similar to its baseline on internal web search data.

How the features are designed

MSLR-WEB30K is the one large set with documented features, and its 136 dimensions read like a syllabus of classical retrieval. The same statistics are computed over five streams of each page (body, anchor text, title, URL and the whole document):

That adds to 136. Three design principles are visible. Per-term statistics are aggregated with min, max, mean and variance, because queries have different lengths and a model needs fixed-width input. Every text-matching score is computed per field, because a match in a title means more than one in body text. And query-independent signals such as link-based authority, quality scores and clicks sit alongside the query-dependent ones, which is what lets a ranker prefer the trustworthy page among several that match equally well. The Yahoo overview describes the same habits: counts normalised into rates such as click-through rate, and host-level counterparts of document features so new pages inherit evidence from their site.

Trees versus neural rankers: the benchmarks disagree with the leaderboards

On feature-vector benchmarks, gradient-boosted trees have been hard to beat. Qin and colleagues showed that most recent neural learning-to-rank models were, by a large margin, worse than the best publicly available GBDT implementations on these datasets, then identified the weaknesses responsible and proposed a framework to address them (Qin et al., 2021, Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?, ICLR).

Text benchmarks such as MS MARCO tell the opposite story, with cross-encoders far ahead of BM25. Istella22 was built to put both on the same test: 8,421,456 documents with text, the same 220 features, and 2,198 text queries with judgments. In its preliminary experiments, LambdaMART on hand-crafted features outperformed state-of-the-art neural text re-rankers such as monoT5 (Dato, MacAvaney, Nardini, Perego & Tonellotto, 2022, The Istella22 Dataset, SIGIR). The likely reading is that the features carry evidence text models cannot see, such as authority, quality and user behaviour, not that trees understand language better.

When it breaks

Behavioural features leak. Click counts and dwell time are strong features, and if they are computed from logs overlapping the period that produced the labels, the model learns to read the answer. Production pipelines need strict time boundaries between feature windows and label windows.

Anonymous features block error analysis. With Yahoo or Istella vectors you cannot ask why a document ranked badly.

Candidate lists come from a retrieval stage. Benchmark documents were chosen by some earlier system, so a model learns to separate relevant from non-relevant within that pool, not across a collection. It inherits the pool's recall ceiling, as in any multi-stage cascade.

Feature cost is not in the data. The benchmarks record values, not the milliseconds each feature takes to compute at serving time, so the best offline model may be unaffordable in the stage it was meant for.

Small differences are noise. When leading systems differ in the third decimal of nDCG, fold-to-fold variance and query sampling can reorder them.