6 posts.
RLHF taught a model to answer one question well. Agentic RL asks a harder thing: take fifty actions in a live environment, most of them invisible in the final reward, and still learn which ones mattered. That single change, from …
An unbiased estimate of every policy's value does not give you an unbiased choice of policy. The moment an optimiser searches over importance-weighted estimates, it goes looking for the estimator's noise, and the history of learn…
Replace the reward model with a function that simply checks the answer, and a frontier reasoning model falls out of pure reinforcement learning. The catch is what 'checkable' quietly assumes, and what the model learns to exploit.
A 7B model that thinks for thirty seconds can outscore a 70B model that answers instantly. Test-time compute is the lever that makes this possible, and it is quietly rewriting the economics of model design.
GRPO weights every token by its own importance ratio, and on long responses that single-sample estimator quietly poisons the gradient until the model collapses. GSPO moves the ratio up to the whole sequence, and Qwen3's largest m…
A thousand identical requests to the same model at temperature zero produced eighty different completions. The cause is not random seeds, not sampling, and not GPU atomics. It is that reduction kernels change their summation orde…