Token Healing and Boundary Bias

Prompt a model with The URL is https: and ask it to continue. It will often produce // followed by something odd, or hesitate in a way it never does with normal text. The reason is not that URLs are hard. It is that : and :// are both tokens, the training corpus contains :// far more often than a bare : followed by /, and by ending the prompt at : you have handed the model a token sequence that essentially never occurs in its training distribution.

This is boundary bias, and it appears anywhere a prompt ends inside what would normally be a single token: mid-word, at a partial identifier, after an opening quote, before a unit suffix.

The mechanism

Tokenisation is greedy and context-dependent. The BPE merge table applied to https:// yields, in most vocabularies, a token for https and a token for ://. Applied to the truncated string https:, it yields https and :. Those are different sequences. The model's next-token distribution conditioned on [https][:] is estimated from the vanishingly few training documents where a colon really did stand alone after https, which are mostly typos and truncated logs.

Formally, the model estimates \(p(x_{t} \mid x_{<t})\) over token sequences, but the user asked a question about character sequences. The map from characters to tokens is not injective in the direction that matters: many token sequences decode to the same string, and the tokeniser picks exactly one of them. When the prompt boundary forces a non-canonical choice, the conditioning is off-distribution even though the text is fine.

Token healing

The fix, implemented in guidance and several inference servers, is to back up. Drop the final token of the prompt, and constrain the first generated token to be one that starts with the dropped token's string. Generation then re-derives the boundary itself, choosing the canonical merge if that is what the distribution prefers.

The procedure in three steps:

1. Tokenise the prompt. Remove the last token \(v\), whose surface string is \(s\).
2. Build the set \(\mathcal{C} = \{u \in V : \text{str}(u) \text{ starts with } s\}\).
3. Sample the next token from the model's distribution restricted to \(\mathcal{C}\), then continue normally.

For https: the dropped token is :, the candidate set contains :, ://, :8080, and similar, and the model picks :// with high probability. The generated text is identical in characters to what naive completion was trying to produce, but the conditioning is now on a sequence the model has actually seen.

Where it matters most

Code completion. Every fill-in-the-middle request ends mid-identifier by construction. A prefix ending at self.get_ forces the tokeniser to split what would have been get_user into pieces the model rarely sees adjacent.

Constrained and structured decoding. Grammar-constrained decoders mask the vocabulary to tokens allowed by the grammar. Because grammars are defined over characters and masks are applied over tokens, a grammar that permits the string true may accidentally forbid the single token true while permitting the pair tr + ue, steering the model onto a low-probability path that satisfies the grammar and degrades quality. This is the same boundary problem wearing a different hat, and it is why grammar-aligned decoding is harder than it looks. See structured output coercion.

Retrieval-augmented prompts. Chunk boundaries that land mid-token produce the same effect at the seam between retrieved context and instruction.

When it breaks

Token healing is cheap but not free, and it has real edge cases. It requires a vocabulary lookup by string prefix, which is a trie over the vocabulary that most serving stacks do not build by default. It interacts badly with prompt caching: removing the last prompt token changes the cache key for the final block, though not for earlier blocks. And it silently changes the semantics of a prompt that deliberately ends at a boundary, such as a few-shot template that ends with a bare : because the examples do too. In that case the non-canonical sequence is exactly what the model was tuned on, and healing makes things worse.

The general lesson generalises past this one trick: the model's units are tokens, the user's units are characters, and every place those two disagree is a place where a system built on the wrong one will be subtly wrong. This is the same root cause behind why LLMs cannot spell and behind arithmetic tokenisation failures.