Training Objectives
Next-token prediction, masked LM, span corruption, fill-in-the-middle, and auxiliary losses.
13concepts
66flashcards
91minutes of reading
- 01 Fill-in-the-Middle A way to teach a plain causal decoder to infill text by rearranging training documents, not the model, so a single architecture serves both left-to-right generation and code-editor-style completion.
- 02 Label Smoothing Softening a one-hot target so the model is never rewarded for driving a correct-class probability all the way to 1, trading a little training loss for calibration and generalisation.
- 03 Loss Masking in Fine-Tuning Supervised fine-tuning uses the same shifted next-token loss as pretraining, computed over the exact same kind of sequence; the only change is that most of the sequence is excluded from the loss entirely.
- 04 Masked Language Modelling BERT's pretraining task hides tokens instead of futures, trading the ability to generate text for representations that read both directions at once.
- 05 Pretraining Objectives Why the loss a model is trained to minimise decides what it can become, and how next-token prediction beat masked and span objectives to own the generative era.
- 06 Sequence Packing and Document Masking Concatenating short documents to fill a fixed context length eliminates padding waste, but only if the attention mask and position IDs are made document-aware; done naively, packing quietly teaches the model to attend across unrelated documents.
- 07 Span Corruption T5's pretraining task hides whole spans instead of single tokens and asks a decoder to generate only the missing pieces, trading BERT's per-token reconstruction for a shorter, cheaper target sequence.