Human preference labels are the most expensive ingredient in a modern aligned model. Constitutional AI replaced most of them with a written document and a model judging itself, and the idea quietly took over the alignment stack.
Replace the reward model with a function that simply checks the answer, and a frontier reasoning model falls out of pure reinforcement learning. The catch is what 'checkable' quietly assumes, and what the model learns to exploit.
DPO is the right default for almost every preference-tuning project in 2026, but the cases where PPO still wins are sharper and more common than the simplicity pitch admits.
You add a KL penalty to an RLHF objective, log it, and it prints minus 0.03. KL divergence is provably non-negative, and nothing is broken. One formula does four different jobs in modern machine learning, and almost every confusi…
GRPO weights every token by its own importance ratio, and on long responses that single-sample estimator quietly poisons the gradient until the model collapses. GSPO moves the ratio up to the whole sequence, and Qwen3's largest m…