Alignment & Post-Training
SFT, reward modelling, DPO/IPO/KTO/ORPO, model merging, and evaluating an aligned model.
21concepts
144flashcards
161minutes of reading
- 01 Building a Preference Dataset A preference dataset pairs model outputs and records which one a human (or AI judge) preferred, providing the training signal that separates a helpful assistant from a raw base model.
- 02 Chat Templates and Special Tokens Chat templates are Jinja2 strings stored in a tokeniser that convert structured message lists into the exact token sequences a fine-tuned model was trained to process.
- 03 DPO in Practice DPO eliminates the separate reward model and RL loop of classic RLHF by reparameterising the reward directly into a classification loss over preferred and rejected response pairs.
- 04 Evaluating an Aligned Model Evaluating an aligned model requires measuring three partially competing properties simultaneously: helpfulness, harmlessness, and honesty, and every method for doing so introduces its own systematic biases.
- 05 KTO: Unpaired Preference Learning KTO aligns language models using only binary good/bad labels per response, avoiding the paired (chosen, rejected) format that makes preference data expensive and brittle to collect.
- 06 Length Bias and Verbosity Control Reward models trained on human preference data systematically score longer outputs higher regardless of quality, causing RLHF-trained models to inflate response length rather than improve content.
- 07 Model Merging: Linear and SLERP Linear and SLERP merging combine the weight tensors of separately fine-tuned models into a single deployable checkpoint, trading off alignment and capability at zero inference cost.
- 08 Model Soups Model soups average the weights of multiple independently fine-tuned checkpoints to produce a single model that outperforms any individual checkpoint without increasing inference cost.
- 09 ORPO and Reference-Free Alignment ORPO collapses supervised fine-tuning and preference alignment into a single training phase by appending a log-odds-ratio penalty directly to the NLL loss, removing the need for a reference model.
- 10 RLAIF and Constitutional Feedback RLAIF replaces human preference labels with an AI judge, and Constitutional AI extends this by encoding a written list of principles so the model critiques and revises its own outputs before any RL training begins.
- 11 Reward Hacking in Alignment Reward hacking occurs when a model maximises its training reward signal through behaviours that violate the designer's intent, undermining alignment despite high measured scores.
- 12 Reward Modelling in Practice A reward model is a learned surrogate for human preference that RLHF uses to provide a differentiable training signal, and its quality determines how aligned the final policy is.
- 13 Supervised Fine-Tuning for Instructions Supervised fine-tuning on curated instruction-response pairs is the first step that transforms a raw pretrained language model into a model that reliably follows human instructions.
- 14 The KL Penalty and Reference Model The KL penalty constrains a fine-tuned language model to stay statistically close to its pre-trained reference, preventing reward hacking while preserving the capabilities built during pretraining.
- 15 The Post-Training Pipeline A structured walkthrough of the four-stage process that converts a raw pretrained language model into a deployable assistant, from supervised fine-tuning through reward modelling, RLHF, and preference-optimisation alternatives.
- 16 DPO and Preference Optimisation How Direct Preference Optimisation collapses the reward-model-plus-PPO pipeline into a single classification loss, and where the RLHF machinery still earns its keep.
- 17 IPO and the Overfitting Fix IPO replaces DPO's sigmoid loss with a squared identity transform, eliminating the theoretical overfitting guarantee that breaks when preference data is finite and deterministic.
- 18 Multi-Objective Alignment Multi-objective alignment trains a single language model to satisfy several competing criteria simultaneously by navigating the Pareto front of reward trade-offs rather than collapsing them into one scalar.
- 19 Online vs Offline Preference Optimisation Offline preference optimisation trains on a fixed dataset of ranked responses, while online methods continuously sample from the current policy, and that single difference has substantial consequences for distribution coverage, reward hacking risk, and final alignment quality.
- 20 PPO for RLHF in Practice A concrete walkthrough of how Proximal Policy Optimisation is wired into the RLHF pipeline, covering the four-model setup, the clipped objective, KL penalty shaping, and the failure modes that kill real training runs.
- 21 TIES and DARE Merging TIES and DARE are two parameter-space merging algorithms that resolve weight interference when combining multiple fine-tuned models into one, avoiding retraining entirely.