Portfolio ↗

Tokenisation

BPE, WordPiece, Unigram, and the ways subword vocabularies quietly shape model behaviour.

11concepts
56flashcards
77minutes of reading
Start with Special and Control Tokens Drill the deck
All levels beginner intermediate advanced
  1. 01 BPE-Dropout and Subword Regularisation A deterministic tokeniser has a blind spot no amount of extra training data fixes on its own, the model never has to be robust to an unfamiliar segmentation of a familiar word, until two techniques deliberately introduced randomness into tokenisation as a training-time fix. advanced 8m 5 cards