Scaling Laws with Vocabulary

Llama 2 used a 32,000-token vocabulary at 7B parameters and at 70B. Llama 3 raised it to 128,256 for every size. Gemma shipped 256,000. These are three different answers to a question that, until recently, nobody had studied the way width, depth, and token count were studied: given a fixed compute budget, how large should the vocabulary be?

The Chinchilla analysis held vocabulary fixed and varied everything else (Hoffmann et al., 2022, arXiv:2203.15556). That is a defensible simplification for a single model family and a bad one across scales, because vocabulary size sits on both sides of the compute equation at once.

Why vocabulary is not free in either direction

Enlarging \(V\) has three simultaneous effects.

It adds parameters: the embedding and unembedding matrices each hold \(V \times d\) entries, which at \(V = 256{,}000\) and \(d = 4096\) is a billion parameters per matrix. Those parameters are non-vocabulary-agnostic; they do no work for tokens that rarely appear.

It shortens sequences. A larger vocabulary raises the average bytes per token, so the same corpus becomes fewer tokens. Fewer tokens at the same FLOPs-per-token means more passes over the data, or a larger effective dataset in bytes for the same token budget. This is the tokenisation tax running in reverse.

It makes each prediction harder and each token more informative. Cross-entropy per token rises mechanically with \(V\) even when the model is better, which is exactly why per-token loss is not comparable across tokenisers and bits per byte is.

Because the second and third effects partially cancel in FLOP terms and the first does not, an optimum exists.

What the measurements say

Tao et al. trained models from 33M to 3B non-vocabulary parameters over vocabularies spanning three orders of magnitude, and estimated the optimum three independent ways: IsoFLOPs curves, a derivative-based estimate of the compute-loss gradient, and a parametric fit (Tao et al., 2024, Scaling Laws with Vocabulary, arXiv:2407.13623, NeurIPS 2024). All three converge on the same shape.

The optimal vocabulary grows with compute, but with a smaller exponent than the non-vocabulary parameter count. Writing \(N_{nv}\) for non-vocabulary parameters, the fitted relationship is approximately

so a model ten times larger wants a vocabulary roughly seven times larger, not ten. Their headline concrete claim is that most released models are under-vocabularised: for a Llama-2-70B-shaped budget the compute-optimal vocabulary is around 216,000 rather than 32,000, and moving to it improves downstream accuracy at equal FLOPs.

The finding is not that bigger is always better. It is that vocabulary should scale, and holding it constant across a model family, which almost every family did before 2024, is guaranteed to be wrong at one end.

The part the scaling law does not capture

Three things fall outside the fitted curve and matter in practice.

Multilinguality is not a scalar. The optimum is computed against a data mixture. A vocabulary sized for English-dominant data is undersized for the same compute if a third of the corpus is Hindi or Japanese, because token fertility differs by a factor of two or more across scripts.

The softmax cost is not just FLOPs. A 256k-way softmax over a long sequence is a memory-bandwidth problem in training and a latency problem in speculative decoding, where the draft model must share the vocabulary. Tao et al. count the FLOPs correctly; wall-clock does not always follow.

Every added token is a candidate for under-training. Growing \(V\) grows the tail, and the tail is where glitch tokens live. Larger vocabularies raise the audit burden, not just the parameter count.

When it breaks

Vocabulary is the least revisable hyperparameter in the stack. Width and depth can be changed by training a new model; vocabulary changes invalidate every tokenised dataset, every KV cache, every stored embedding, and every downstream model that shares the tokeniser for speculative decoding. The scaling law is most valuable before the first pretraining run, and nearly useless after, which is why tokeniser transplantation exists as a separate discipline.