Tokenisation
BPE, WordPiece, Unigram, and the ways subword vocabularies quietly shape model behaviour.
11concepts
56flashcards
77minutes of reading
- 01 Byte-Level BPE Why GPT-2 and its descendants tokenise raw bytes instead of characters, and how that single design choice guarantees any input string can be represented without an unknown-token fallback.
- 02 Detokenisation and Streaming Boundaries Streaming a response token by token can briefly render a garbled glyph on screen, a direct, visible consequence of a multi-byte character being split across more than one token, and a correctness problem, not just a UI quirk.
- 03 The Tokenisation Tax Two prompts with the same meaning can cost a different number of tokens under the same vocabulary, and that gap compounds into real dollars, real context budget, and real latency before the model reasons about anything at all.
- 04 Token Fertility and Multilingual Fairness The same paragraph of news text can tokenise into several times as many tokens in one language as another under a shared vocabulary, a structural cost baked in before any user sends a request, not a rounding error.
- 05 Tokenisation and Arithmetic A model can be strong at multi-step reasoning and still botch four-digit addition, not from a lack of arithmetic ability but because its tokeniser handed it two structurally unrelated token sequences for two numbers that differ by one.
- 06 Unigram LM and SentencePiece A tokeniser that builds its vocabulary top-down instead of bottom-up, assigns every segmentation of a string a real probability, and the library that made it (and BPE) usable without a language-specific pre-tokeniser.
- 07 WordPiece Tokenisation BERT's tokeniser looks like BPE on the surface, but the merge criterion it optimises is different, and that difference is why WordPiece pieces tend to track real morphemes more closely.