Under-Trained and Glitch Tokens

In February 2023 someone asked GPT-3.5 to repeat the string SolidGoldMagikarp back to them. The model said "distribute". Asked again, it produced insults, refusals, and text about topics with no connection to the prompt. The token was in the vocabulary, so the tokeniser encoded it happily into a single id, but the model had almost never seen that id during pretraining. Its embedding row was still close to where initialisation left it, and feeding a near-random vector into layer zero produces near-arbitrary behaviour downstream.

This is not a curiosity. Every model whose tokeniser was trained on a different corpus than its weights carries a set of these tokens, and they are reachable by any user who can type.

Why the vocabulary and the weights disagree

A tokeniser is fit first, usually by BPE or Unigram over a large sample of text, and then frozen. Training data is then filtered, deduplicated, and reweighted, sometimes aggressively. Anything the tokeniser learned from text that the filters later removed keeps its slot in the vocabulary and loses its gradient signal.

The observed sources are mundane and repeatable (Land & Bartolo, 2024, Fishing for Magikarp, arXiv:2405.05417):

• Reddit usernames and forum handles that survived tokeniser fitting and were then deduplicated away.
• Boilerplate scraped from a single crawled site: navigation strings, JavaScript fragments, CDN paths.
• Byte sequences from encodings the final mix barely contains, plus fragments of other scripts.
• Reserved and special-purpose ids added after the fact and never used in the training format.

The result is a vocabulary row whose embedding is untouched by learning. Land and Bartolo found such tokens in every major model family they audited, including GPT-2, GPT-4's cl100k_base, Llama 2, Mistral, and Gemma.

Finding them from the weights

The useful part of that work is that you do not need the training corpus to detect them. Two signals do most of the job.

The first is the unembedding-based indicator. For an under-trained token \(t\), the model has never had to push probability onto its output row, so the row sits near the mean of the unembedding matrix. Compute

where \(U_t\) is the token's row in the unembedding matrix and \(\bar{U}\) the mean row over the vocabulary. Tokens whose row is unusually close to the mean direction, and whose predicted probability under a "repeat this token" prompt is near zero, are candidates.

The second is a verification prompt. Ask the model to echo the token inside a delimiter. A trained token round-trips; an under-trained one produces an unrelated word, an empty string, or a refusal. The combination of a cheap weight-space filter and an expensive prompt-space confirmation is what turns anecdote into an audit.

Why this is a safety problem, not just a bug

Three consequences follow, and each has been observed in production systems.

Prompt-space attack surface. A string that reliably derails a model's output distribution is a jailbreak primitive. It costs the attacker nothing to include, is invisible in a rendered UI when the token is whitespace or a control sequence, and does not resemble any pattern a moderation classifier was trained to catch.

Silent retrieval corruption. Documents containing glitch tokens embed to nonsense vectors. In a RAG corpus, a page of scraped navigation boilerplate can land in a region of embedding space no real query occupies, or worse, in a dense cluster that matches everything weakly.

Fine-tuning does not fix it. A fine-tune touches only the tokens present in the fine-tuning data. Under-trained rows stay under-trained, and if the fine-tune uses a small LoRA on attention projections only, the embedding matrix is not updated at all.

When it breaks

The mitigations are all imperfect and each costs something. Removing the token from the vocabulary changes the tokeniser, which invalidates every cached KV prefix and every stored embedding. Reinitialising the row to the vocabulary mean at least makes the failure boring rather than adversarial, but it does not make the token mean anything. Filtering the inputs pushes the problem to the boundary, where you must decide what to do with a user who legitimately typed a rare identifier.

The durable fix is process, not patching: fit the tokeniser on the post-filter corpus, then audit the trained checkpoint with the unembedding indicator before release, and publish the list. This connects directly to byte-level BPE, which bounds the damage by making every byte sequence representable, and to special and control tokens, which are the most common source of reserved-but-untrained ids.