Cross-Lingual Embedding Alignment

Because nothing in the objective requires it. Masked language modelling predicts tokens within a language, so the model is free to give each language its own region of the space, and incidental alignment from shared subwords is weak.

Empirically representations cluster by language before they cluster by meaning.

The retrieval failure that follows is the worst kind: a Hindi query returns fluent, topically plausible Hindi documents while ignoring the correct English answer, so the system looks like it is working.

Procrustes: learn an orthogonal \(W\) so \(Wx_{\text{src}} \approx x_{\text{tgt}}\) over a seed dictionary. Cheap, interpretable, and assumes the two monolingual spaces are isomorphic up to rotation, which breaks down for typologically distant and low-resource languages.

Translation ranking: train contrastively with parallel sentences as positives, making alignment the training signal rather than a post-processing step. LaBSE combines this with masked LM, translation LM and additive-margin softmax, reaching 83.7% Tatoeba bitext accuracy across 112 languages against a prior 65.5%.

Because translation ranking teaches translation equivalence, which is a narrower relation than relevance. The model learns that a sentence and its translation are close; it does not learn that a passage answers a question asked in another language.

Bitext benchmarks like Tatoeba measure the first relation directly, so they flatter models trained on it.

Always evaluate the relation you actually need. A high Tatoeba score is evidence about alignment, not about retrieval.

At fixed capacity, adding languages eventually degrades per-language quality, because parameters are shared and each language competes for them.

Data imbalance compounds it. Web corpora are dominated by a few high-resource languages, so contrastive training sees orders of magnitude more English pairs than Swahili pairs, and the aligned space is correspondingly sharper in English.

Token fertility adds a third penalty: a language that tokenises into more tokens per word fits less content per context window and costs more per query.

Because a residual language component survives alignment: the space is shared but not perfectly so, and that component acts as a common offset shared by all documents in a language, which raises their similarity to any query in that language.

Per-language mean subtraction on the index removes much of it, and is rarely done because it requires reliable language labels on every document.

The same operation helps with anisotropy and hubness, since all three are consequences of an uncentred common component.

Because per-language task counts drop sharply in the tail. MMTEB's 250+ language coverage is a genuine advance, but a language near the tail may be represented by a single dataset, so its score has no error bar worth the name and reflects that dataset's domain rather than the language.

Code-mixed and transliterated text (Hinglish, romanised Arabic) is worse still: it is off-distribution for both language identification and alignment, and behaves far below the headline number.

Because nothing in the objective requires it. Masked language modelling predicts tokens within a language, so the model is free to give each language its own region of the space, and incidental alignment from shared subwords is weak.

Empirically representations cluster by language before they cluster by meaning.

The retrieval failure that follows is the worst kind: a Hindi query returns fluent, topically plausible Hindi documents while ignoring the correct English answer, so the system looks like it is working.

Procrustes: learn an orthogonal \(W\) so \(Wx_{\text{src}} \approx x_{\text{tgt}}\) over a seed dictionary. Cheap, interpretable, and assumes the two monolingual spaces are isomorphic up to rotation, which breaks down for typologically distant and low-resource languages.

Translation ranking: train contrastively with parallel sentences as positives, making alignment the training signal rather than a post-processing step. LaBSE combines this with masked LM, translation LM and additive-margin softmax, reaching 83.7% Tatoeba bitext accuracy across 112 languages against a prior 65.5%.

Because translation ranking teaches translation equivalence, which is a narrower relation than relevance. The model learns that a sentence and its translation are close; it does not learn that a passage answers a question asked in another language.

Bitext benchmarks like Tatoeba measure the first relation directly, so they flatter models trained on it.

Always evaluate the relation you actually need. A high Tatoeba score is evidence about alignment, not about retrieval.

At fixed capacity, adding languages eventually degrades per-language quality, because parameters are shared and each language competes for them.

Data imbalance compounds it. Web corpora are dominated by a few high-resource languages, so contrastive training sees orders of magnitude more English pairs than Swahili pairs, and the aligned space is correspondingly sharper in English.

Token fertility adds a third penalty: a language that tokenises into more tokens per word fits less content per context window and costs more per query.

Because a residual language component survives alignment: the space is shared but not perfectly so, and that component acts as a common offset shared by all documents in a language, which raises their similarity to any query in that language.

Per-language mean subtraction on the index removes much of it, and is rarely done because it requires reliable language labels on every document.

The same operation helps with anisotropy and hubness, since all three are consequences of an uncentred common component.

Because per-language task counts drop sharply in the tail. MMTEB's 250+ language coverage is a genuine advance, but a language near the tail may be represented by a single dataset, so its score has no error bar worth the name and reflects that dataset's domain rather than the language.

Code-mixed and transliterated text (Hinglish, romanised Arabic) is worse still: it is off-distribution for both language identification and alignment, and behaves far below the headline number.