Training Objectives
Next-token prediction, masked LM, span corruption, fill-in-the-middle, and auxiliary losses.
14concepts
76flashcards
98minutes of reading
- 01 MoE Load-Balancing Loss Without an explicit penalty, a mixture-of-experts router collapses onto a handful of favourite experts within the first few hundred steps; the auxiliary load-balancing loss is the mechanism that stops it.
- 02 Multi-Token Prediction Training a language model to predict the next n tokens at once from a shared trunk, which densifies the training signal and hands you free draft heads for self-speculative decoding.
- 03 The Softmax Cross-Entropy Gradient The combined derivative of softmax and cross-entropy collapses to one subtraction, predicted probability minus true label, and that simplification is why every production training loop fuses the two ops instead of computing them separately.
- 04 Z-Loss and Logit Regularisation Softmax is shift-invariant, which leaves a direction in logit space that cross-entropy never penalises; z-loss closes that gap and is what keeps large-scale bf16 training from spiking.