A support search drops any result scoring below 0.75 cosine similarity. The team re-embeds the whole corpus with a newer embedding model and rebuilds the index; now the same cutoff throws away almost every result. What is the mistake?

What is being tested

Whether you treat a similarity score as a ranking signal rather than a probability. Nearly every team that builds retrieval invents an absolute cutoff in week one, and that cutoff is the first thing that breaks when anything upstream changes.

The mechanism

Cosine similarity is the angle between two vectors in a space whose shape was decided by the training objective, not by any shared convention. Contrastive training pushes text into a narrow cone, so on some models two completely unrelated English sentences still score around 0.7, because every vector points roughly the same way. Another model, trained with a different objective or temperature, spreads the same pairs down to 0.05.

The number has no meaning outside one model paired with one corpus. The same 0.75 keeps nearly everything on the first model and nearly nothing on the second, which is exactly the symptom in the stem. The corpus matters too: a cutoff tuned on marketing pages will be wrong on API reference pages, because the second corpus is lexically narrower and every chunk looks similar to every query.

Why the other options fail

• "The new model is weaker." The plausible reading, and the one that gets a good upgrade rolled back. Ranking quality is measured by whether the right passage appears in the top k, not by the magnitude of its score. Measure recall at 10 on a labelled query set before blaming the model; the new model may rank better while scoring lower.
• "The vectors are not normalised." A real bug, and worth checking once, but it does not produce this pattern. Un-normalised vectors make cosine and dot product disagree and distort ranking; here the ranking is fine and only the absolute values moved.
• "The index was never rebuilt." The single most common retrieval outage, since vectors from two different models are not comparable at all. The stem rules it out, and it would show as nonsense results rather than as good results below a threshold.

How to set a cutoff that survives

1. Calibrate against your own corpus. Sample roughly 500 random query-document pairs that you know are unrelated, score them, and take the 99^th percentile of that distribution as the floor. Anything below it is indistinguishable from noise. The whole calibration costs one batch of embeddings and runs in about 200 ms of scoring; a threshold that has been in production for a year without one is a number nobody can defend.
2. Store the threshold beside the embedding model version, and re-run the calibration job as a step in every re-embedding pipeline, so the number cannot outlive the model it was derived from.
3. Prefer top-k with an abstention rule. Retrieve a fixed k, rerank, and let the generator say it does not know when the reranked top result is weak. A relative decision needs no portable constant.

Common weak answers

• "Lower the threshold to 0.5." It works until the next model change, and it silently lets noise into the context window, which produces confident wrong answers rather than empty ones.
• "Use Euclidean distance instead." Same problem in a different unit. The scale is still model-specific.
• "Tune it on a handful of queries." Ten queries cannot separate a real shift from sampling noise; the calibration needs hundreds of pairs to be worth anything.