Alignment & Post-Training
SFT, reward modelling, DPO/IPO/KTO/ORPO, model merging, and evaluating an aligned model.
21concepts
144flashcards
161minutes of reading
- 01 Building a Preference Dataset A preference dataset pairs model outputs and records which one a human (or AI judge) preferred, providing the training signal that separates a helpful assistant from a raw base model.
- 02 Chat Templates and Special Tokens Chat templates are Jinja2 strings stored in a tokeniser that convert structured message lists into the exact token sequences a fine-tuned model was trained to process.
- 03 DPO in Practice DPO eliminates the separate reward model and RL loop of classic RLHF by reparameterising the reward directly into a classification loss over preferred and rejected response pairs.
- 04 Evaluating an Aligned Model Evaluating an aligned model requires measuring three partially competing properties simultaneously: helpfulness, harmlessness, and honesty, and every method for doing so introduces its own systematic biases.
- 05 KTO: Unpaired Preference Learning KTO aligns language models using only binary good/bad labels per response, avoiding the paired (chosen, rejected) format that makes preference data expensive and brittle to collect.
- 06 Length Bias and Verbosity Control Reward models trained on human preference data systematically score longer outputs higher regardless of quality, causing RLHF-trained models to inflate response length rather than improve content.
- 07 Model Merging: Linear and SLERP Linear and SLERP merging combine the weight tensors of separately fine-tuned models into a single deployable checkpoint, trading off alignment and capability at zero inference cost.
- 08 Model Soups Model soups average the weights of multiple independently fine-tuned checkpoints to produce a single model that outperforms any individual checkpoint without increasing inference cost.
- 09 ORPO and Reference-Free Alignment ORPO collapses supervised fine-tuning and preference alignment into a single training phase by appending a log-odds-ratio penalty directly to the NLL loss, removing the need for a reference model.
- 10 RLAIF and Constitutional Feedback RLAIF replaces human preference labels with an AI judge, and Constitutional AI extends this by encoding a written list of principles so the model critiques and revises its own outputs before any RL training begins.
- 11 Reward Hacking in Alignment Reward hacking occurs when a model maximises its training reward signal through behaviours that violate the designer's intent, undermining alignment despite high measured scores.
- 12 Reward Modelling in Practice A reward model is a learned surrogate for human preference that RLHF uses to provide a differentiable training signal, and its quality determines how aligned the final policy is.
- 13 Supervised Fine-Tuning for Instructions Supervised fine-tuning on curated instruction-response pairs is the first step that transforms a raw pretrained language model into a model that reliably follows human instructions.
- 14 The KL Penalty and Reference Model The KL penalty constrains a fine-tuned language model to stay statistically close to its pre-trained reference, preventing reward hacking while preserving the capabilities built during pretraining.
- 15 The Post-Training Pipeline A structured walkthrough of the four-stage process that converts a raw pretrained language model into a deployable assistant, from supervised fine-tuning through reward modelling, RLHF, and preference-optimisation alternatives.