Hard Negative Mining and Contrastive Embedding Training
A retrieval embedder is only as good as the negatives it was trained against, and the gap between easy in-batch negatives and mined hard negatives is the single largest lever in dual-encoder training.
Train a dual encoder with random negatives and it learns to separate a question about renal function from a passage about football fixtures. That is not the task. At inference the retriever must separate that question from ten thousand other medical passages, all of which are topically adjacent. The training distribution and the test distribution disagree, and the model optimises for the wrong one.
Fixing that mismatch is most of the craft in embedding training.
The objective
Dual encoders are trained with InfoNCE over a positive and a set of negatives. For query \(q\) with positive \(d^{+}\) and negatives \(d^{-}_{j}\):
where \(s\) is usually a dot product on L2-normalised vectors and \(\tau\) is a temperature. The gradient pushes the query toward its positive and away from each negative, weighted by how much probability mass that negative currently holds. A negative the model already scores near zero contributes almost no gradient. That single fact explains why negative selection dominates.
Three generations of negatives
In-batch negatives. Treat every other document in the batch as a negative for every query. It is nearly free: the similarity matrix is one matmul you already computed. Dense Passage Retrieval used this to beat BM25 top-20 accuracy by 9 to 19 points absolute (Karpukhin et al., 2020, Dense Passage Retrieval for Open-Domain QA, EMNLP, arXiv:2004.04906). Its limitation is structural: batch members are random draws from the corpus, so almost all of them are trivially separable, and the effective number of informative negatives per step is small regardless of batch size.
BM25-mined negatives. Retrieve top lexical matches for the query and use the non-relevant ones. Cheap, static, and much harder than random. The failure mode is that they are hard in a lexical sense, so the model learns to be different from BM25 rather than to be good.
ANN-mined negatives from the model itself. Periodically index the corpus with the current checkpoint, retrieve top-k for each training query, and use those as negatives. These are hard by the model's own definition, which is exactly the distribution it faces at test time. ANCE formalised this with an index refreshed asynchronously during training, and reported that the resulting model outperformed all competitive dense and sparse baselines, nearly matching sparse-retrieval-plus-BERT-reranking at close to 100x lower latency (Xiong et al., 2020, Approximate Nearest Neighbor Negative Contrastive Learning, arXiv:2007.00808).
The false negative problem
Mined negatives are top-ranked documents that are not labelled relevant. In any real corpus, some of them are relevant and simply unlabelled. Training then explicitly pushes the query away from a correct answer.
This is the central tension: the harder you mine, the higher the proportion of your negatives that are actually positives. Mitigations in practice are a denoising threshold (discard negatives scoring above some similarity to the positive), a cross-encoder or LLM filter over mined candidates, and sampling from a rank window such as ranks 30 to 100 rather than the top 10. None removes the problem; they trade recall of hard negatives against label noise.
Scale versus difficulty
There are two routes to a strong contrastive signal, and they are partly substitutes.
E5 took the scale route: weakly-supervised contrastive pretraining on a large curated pair corpus (CCPairs), then fine-tuning. It became the first model to beat BM25 on BEIR zero-shot without labelled data, and when fine-tuned beat embedding models with 40x more parameters (Wang et al., 2022, Text Embeddings by Weakly-Supervised Contrastive Pre-training, arXiv:2212.03533).
Huge batches make in-batch negatives less trivially easy simply by sampling more of the corpus per step, which is why contrastive training pushes batch sizes into the tens of thousands with gradient caching. But batch size buys negatives that are more numerous, not more informative, and it saturates. Mining buys informativeness directly.
When it breaks
Refreshing the index is the expensive part. ANN-mined negatives require re-encoding and re-indexing the corpus periodically. On a large corpus this can dominate training cost, and stale indexes produce negatives the model has already learned to reject.
Temperature interacts with negative hardness. Low \(\tau\) sharpens the softmax so the hardest negative dominates the gradient. Combine low temperature with aggressively mined negatives and a single mislabelled false negative can swamp a step. Tuning them independently is a mistake.
Hard negatives can degrade non-retrieval tasks. An embedder tuned to separate near-duplicates makes a poor clustering or semantic-similarity model, because those tasks want near-duplicates close together. Retrieval-optimised embeddings are not general-purpose, whatever a single leaderboard number suggests; see embedding benchmarks and the zero-shot problem.
Mining amplifies training-set bias. If your labelled positives over-represent a domain, mined negatives concentrate there too, and the model becomes sharply better in that domain and quietly worse elsewhere.
12 flashcards for this concept
Click a card to reveal the answer.