Tokenisation
BPE, WordPiece, Unigram, and the ways subword vocabularies quietly shape model behaviour.
15concepts
136flashcards
108minutes of reading
- 01 Special and Control Tokens The handful of reserved vocabulary entries no amount of BPE merging could ever produce from ordinary text, and why keeping them outside the mergeable vocabulary is a real security boundary, not just bookkeeping.
- 02 Tokenisation and BPE How models turn raw text into integer tokens, and why the vocabulary choice silently shapes model behaviour.
- 03 Why LLMs Cannot Spell or Count Letters The classic "how many r's in strawberry" failure is not a reasoning gap, it is a representational one, caused by the model never seeing the individual letters that make up a compressed token.