Embedding Benchmarks and the Zero-Shot Problem

It is \(z = 1 - n_{\text{train}}/n_{\text{total}}\), where \(n_{\text{train}}\) counts the benchmark datasets the model was trained on. The documented example is e5-mistral-7b-instruct at 95% on MTEB (English, v2), so it trained on roughly 5% of the benchmark's training splits.

It matters because MTEB published training splits for many of its datasets and models began training on them. A model that trained on the NQ training split is not evaluating zero-shot on an NQ-derived task, so its score stops predicting behaviour on your corpus.

Read the zero-shot column before the score column.

Because it punished transparency. Filtering to 100% zero-shot excluded models whose labs disclosed their training data while leaving in models whose data was undisclosed and therefore unverifiable.

The result would have been a leaderboard biased toward secrecy.

Reporting the score alongside rank preserves the signal without creating that incentive, at the cost of requiring the reader to actually look at it.

Its premise was that retrieval quality is an out-of-distribution question: models were being evaluated in-domain, which said nothing about transfer. BEIR assembled 18 heterogeneous datasets and evaluated 10 systems zero-shot.

The finding that stuck is that BM25 is a robust baseline many dense retrievers failed to beat out of domain, while reranking and late-interaction models achieved the best zero-shot quality at markedly higher compute cost.

That result is the reason hybrid retrieval remains a default rather than a legacy compromise.

It averages task families with incompatible geometric requirements: retrieval, reranking, clustering, classification, semantic similarity, pair classification, summarisation, bitext mining. A high average is a compromise across them, and MTEB's own launch finding was that no method dominates across tasks.

Also absent from the average: your domain (legal, clinical and code are barely represented), your chunk length (many benchmark documents are short), and every cost dimension, since dimensionality, sequence limit and inference price do not appear in the rank at all.

That it is not the axis. Across 500+ tasks in 250+ languages, the best publicly available model was multilingual-e5-large-instruct at 560 million parameters, well below the largest entrants (Enevoldsen et al., ICLR 2025).

Embedding quality is dominated by training data and objective, not parameter count, which is unlike the scaling story for generative models and frequently surprises teams who assume a 7B embedder must beat a 0.5B one.

Reproduce the ranking on your own data. A few hundred labelled query-document pairs from production traffic, scored for recall at k against three or four candidate models, costs an afternoon and routinely reverses the leaderboard order.

Check the zero-shot scores of the models involved, since an in-domain-trained model tends to lose most of its lead off-domain. Check chunk length and domain overlap too.

Leaderboards are a shortlist generator, not a decision procedure.

It is \(z = 1 - n_{\text{train}}/n_{\text{total}}\), where \(n_{\text{train}}\) counts the benchmark datasets the model was trained on. The documented example is e5-mistral-7b-instruct at 95% on MTEB (English, v2), so it trained on roughly 5% of the benchmark's training splits.

It matters because MTEB published training splits for many of its datasets and models began training on them. A model that trained on the NQ training split is not evaluating zero-shot on an NQ-derived task, so its score stops predicting behaviour on your corpus.

Read the zero-shot column before the score column.

Because it punished transparency. Filtering to 100% zero-shot excluded models whose labs disclosed their training data while leaving in models whose data was undisclosed and therefore unverifiable.

The result would have been a leaderboard biased toward secrecy.

Reporting the score alongside rank preserves the signal without creating that incentive, at the cost of requiring the reader to actually look at it.

Its premise was that retrieval quality is an out-of-distribution question: models were being evaluated in-domain, which said nothing about transfer. BEIR assembled 18 heterogeneous datasets and evaluated 10 systems zero-shot.

The finding that stuck is that BM25 is a robust baseline many dense retrievers failed to beat out of domain, while reranking and late-interaction models achieved the best zero-shot quality at markedly higher compute cost.

That result is the reason hybrid retrieval remains a default rather than a legacy compromise.

It averages task families with incompatible geometric requirements: retrieval, reranking, clustering, classification, semantic similarity, pair classification, summarisation, bitext mining. A high average is a compromise across them, and MTEB's own launch finding was that no method dominates across tasks.

Also absent from the average: your domain (legal, clinical and code are barely represented), your chunk length (many benchmark documents are short), and every cost dimension, since dimensionality, sequence limit and inference price do not appear in the rank at all.

That it is not the axis. Across 500+ tasks in 250+ languages, the best publicly available model was multilingual-e5-large-instruct at 560 million parameters, well below the largest entrants (Enevoldsen et al., ICLR 2025).

Embedding quality is dominated by training data and objective, not parameter count, which is unlike the scaling story for generative models and frequently surprises teams who assume a 7B embedder must beat a 0.5B one.

Reproduce the ranking on your own data. A few hundred labelled query-document pairs from production traffic, scored for recall at k against three or four candidate models, costs an afternoon and routinely reverses the leaderboard order.

Check the zero-shot scores of the models involved, since an in-domain-trained model tends to lose most of its lead off-domain. Check chunk length and domain overlap too.

Leaderboards are a shortlist generator, not a decision procedure.