Alignment & Post-Training
SFT, reward modelling, DPO/IPO/KTO/ORPO, model merging, and evaluating an aligned model.
21concepts
144flashcards
161minutes of reading
- 01 DPO and Preference Optimisation How Direct Preference Optimisation collapses the reward-model-plus-PPO pipeline into a single classification loss, and where the RLHF machinery still earns its keep.
- 02 IPO and the Overfitting Fix IPO replaces DPO's sigmoid loss with a squared identity transform, eliminating the theoretical overfitting guarantee that breaks when preference data is finite and deterministic.
- 03 Multi-Objective Alignment Multi-objective alignment trains a single language model to satisfy several competing criteria simultaneously by navigating the Pareto front of reward trade-offs rather than collapsing them into one scalar.
- 04 Online vs Offline Preference Optimisation Offline preference optimisation trains on a fixed dataset of ranked responses, while online methods continuously sample from the current policy, and that single difference has substantial consequences for distribution coverage, reward hacking risk, and final alignment quality.
- 05 PPO for RLHF in Practice A concrete walkthrough of how Proximal Policy Optimisation is wired into the RLHF pipeline, covering the four-model setup, the clipped objective, KL penalty shaping, and the failure modes that kill real training runs.
- 06 TIES and DARE Merging TIES and DARE are two parameter-space merging algorithms that resolve weight interference when combining multiple fine-tuned models into one, avoiding retraining entirely.