Tokenisation
BPE, WordPiece, Unigram, and the ways subword vocabularies quietly shape model behaviour.
11concepts
56flashcards
77minutes of reading
- 01 Special and Control Tokens The handful of reserved vocabulary entries no amount of BPE merging could ever produce from ordinary text, and why keeping them outside the mergeable vocabulary is a real security boundary, not just bookkeeping.
- 02 Tokenisation and BPE How models turn raw text into integer tokens, and why the vocabulary choice silently shapes model behaviour.
- 03 Why LLMs Cannot Spell or Count Letters The classic "how many r's in strawberry" failure is not a reasoning gap, it is a representational one, caused by the model never seeing the individual letters that make up a compressed token.
- 04 Byte-Level BPE Why GPT-2 and its descendants tokenise raw bytes instead of characters, and how that single design choice guarantees any input string can be represented without an unknown-token fallback.
- 05 Detokenisation and Streaming Boundaries Streaming a response token by token can briefly render a garbled glyph on screen, a direct, visible consequence of a multi-byte character being split across more than one token, and a correctness problem, not just a UI quirk.
- 06 The Tokenisation Tax Two prompts with the same meaning can cost a different number of tokens under the same vocabulary, and that gap compounds into real dollars, real context budget, and real latency before the model reasons about anything at all.
- 07 Token Fertility and Multilingual Fairness The same paragraph of news text can tokenise into several times as many tokens in one language as another under a shared vocabulary, a structural cost baked in before any user sends a request, not a rounding error.
- 08 Tokenisation and Arithmetic A model can be strong at multi-step reasoning and still botch four-digit addition, not from a lack of arithmetic ability but because its tokeniser handed it two structurally unrelated token sequences for two numbers that differ by one.
- 09 Unigram LM and SentencePiece A tokeniser that builds its vocabulary top-down instead of bottom-up, assigns every segmentation of a string a real probability, and the library that made it (and BPE) usable without a language-specific pre-tokeniser.
- 10 WordPiece Tokenisation BERT's tokeniser looks like BPE on the surface, but the merge criterion it optimises is different, and that difference is why WordPiece pieces tend to track real morphemes more closely.
- 11 BPE-Dropout and Subword Regularisation A deterministic tokeniser has a blind spot no amount of extra training data fixes on its own, the model never has to be robust to an unfamiliar segmentation of a familiar word, until two techniques deliberately introduced randomness into tokenisation as a training-time fix.