Tokeniser Transplantation

An English-centric model asked to process Telugu spends four to six times as many tokens per character as it does on English. The model may know Telugu perfectly well; the tokeniser does not. Every request costs more, the effective context window shrinks by the same factor, and latency scales with it. Retraining the model on a better tokeniser costs the full pretraining budget. Transplanting one costs a few billion tokens at most, and sometimes zero.

The observation that makes this possible: a transformer's tokeniser touches the model in exactly two places, the input embedding matrix \(E \in \mathbb{R}^{V \times d}\) and the output unembedding \(U \in \mathbb{R}^{V \times d}\). Everything between them, which is where all the linguistic competence lives, is vocabulary-agnostic.

Four ways to fill a new embedding matrix

Overlap copy plus random init. Keep the rows for tokens present in both vocabularies, initialise the rest from the mean and covariance of the old matrix, then continue pretraining. Simple, and a strong baseline when the vocabularies overlap heavily. Dagan et al. found that switching a code tokeniser on an already-pretrained Llama and continuing for around 50 billion tokens recovered full performance, and that shorter continuations left a measurable gap (Dagan et al., 2024, arXiv:2402.01035).

Auxiliary-embedding alignment. WECHSEL initialises each new token's embedding as a similarity-weighted average of old embeddings, where the similarity comes from static word vectors in both languages plus a bilingual dictionary (Minixhofer et al., 2022, WECHSEL, NAACL 2022). The reported effect was up to 64 times less training effort than pretraining from scratch at comparable quality. FOCUS is the same idea driven by overlapping-token statistics rather than external dictionaries.

Decomposition. Initialise a new token's embedding from the embeddings of the old tokens it decomposes into. For a new token _transplant, average or sum the old model's _trans, pl, ant rows. It costs nothing, needs no external resources, and works surprisingly well for morphologically transparent vocabularies.

Hypernetwork prediction. Train a network that takes a tokeniser as input and predicts the embedding matrix for it. Zero-Shot Tokenizer Transfer does this, reaching close to original performance on cross-lingual and coding tasks with no gradient steps on the target model at all, and closing the remaining gap with under a billion tokens of continued training (Minixhofer et al., 2024, Zero-Shot Tokenizer Transfer, arXiv:2405.07883, NeurIPS 2024). The cost is training the hypernetwork once per base model family.

What actually gets recovered, and when

The pattern across all four families is the same: perplexity spikes immediately after transplantation, then falls fast, and the rate of recovery depends on how much of the model's competence was entangled with the old segmentation. Surface-level tasks recover in hundreds of millions of tokens. Tasks that depend on precise token-level conventions, arithmetic and code formatting in particular, take much longer, because the model learned digit-grouping and indentation habits that were properties of the old merge table.

A useful diagnostic before committing: measure fertility on your target corpus under both tokenisers. If the new tokeniser does not cut tokens-per-character by at least 20 to 25 percent, the transplant is unlikely to pay back its continued-pretraining cost.

When it breaks

Transplantation quietly invalidates a lot of infrastructure. Every cached KV prefix is dead. Every stored document embedding computed by the same encoder is dead. A speculative decoding draft model must share the target's vocabulary exactly, so transplanting the target orphans the draft. Any downstream fine-tune of the original checkpoint cannot be merged into the transplanted one, because the LoRA deltas for \(E\) and \(U\) no longer have matching row spaces.

There is also a failure mode that looks like success. A transplanted model can regain its benchmark scores while losing calibration, because the output distribution over a new vocabulary has not been re-tuned even though argmax accuracy has recovered. Check calibration explicitly after a transplant rather than assuming accuracy carried it.