Tokenisation
BPE, WordPiece, Unigram, and the ways subword vocabularies quietly shape model behaviour.
15concepts
136flashcards
108minutes of reading
- 01 BPE-Dropout and Subword Regularisation A deterministic tokeniser has a blind spot no amount of extra training data fixes on its own, the model never has to be robust to an unfamiliar segmentation of a familiar word, until two techniques deliberately introduced randomness into tokenisation as a training-time fix.
- 02 Scaling Laws with Vocabulary Vocabulary size is a scaling parameter like width and depth, most models pick it too small, and the compute-optimal value grows with model size but more slowly than the parameter count.
- 03 Tokeniser Transplantation How to give a pretrained model a different tokeniser without retraining it, why the embedding matrix is the only thing that must change, and what the four families of methods cost.
- 04 Under-Trained and Glitch Tokens Why some tokens in a model's vocabulary were almost never seen during training, what happens when a user types one, and how to find them from the weights alone.