Training Objectives
Next-token prediction, masked LM, span corruption, fill-in-the-middle, and auxiliary losses.
13concepts
66flashcards
91minutes of reading
- 01 MoE Load-Balancing Loss Without an explicit penalty, a mixture-of-experts router collapses onto a handful of favourite experts within the first few hundred steps; the auxiliary load-balancing loss is the mechanism that stops it.
- 02 The Softmax Cross-Entropy Gradient The combined derivative of softmax and cross-entropy collapses to one subtraction, predicted probability minus true label, and that simplification is why every production training loop fuses the two ops instead of computing them separately.
- 03 Z-Loss and Logit Regularisation Softmax is shift-invariant, which leaves a direction in logit space that cross-entropy never penalises; z-loss closes that gap and is what keeps large-scale bf16 training from spiking.