Training Objectives
Next-token prediction, masked LM, span corruption, fill-in-the-middle, and auxiliary losses.
13concepts
66flashcards
91minutes of reading
- 01 Next-Token Prediction and Cross-Entropy Loss The single objective behind every LLM - predict the next token - and the cross-entropy loss that turns "predict well" into a gradient the model can descend.
- 02 Softmax and Logits How a model's raw output scores become a probability distribution, why the exponential matters, and the numerical trick that keeps softmax from overflowing.
- 03 The Causal LM Pretraining Task How a corpus of raw documents becomes millions of training examples under one masking rule, and why "causal" describes a data and attention decision, not just a loss.