Process Reward Models and Verifiable Rewards
Why scoring every step of a reasoning trace beats scoring only the final answer, and how Ai2 and DeepSeek replaced PRMs entirely with programmatic correctness checks.
Standard RLHF rewards the final answer. For multi-step reasoning that is a brutally sparse signal: the model gets a single scalar at the end of a 50-step chain, with no way to tell which step was the load-bearing mistake. Process reward models (PRMs) score each intermediate step. Reinforcement Learning with Verifiable Rewards (RLVR) sidesteps the reward model entirely and uses a programmatic checker. Both target the same problem - the credit-assignment crisis on long reasoning traces - and they are now the two dominant recipes behind the 2024-2025 reasoning wave.
ORM vs PRM
An Outcome Reward Model (ORM) takes a (prompt, full chain, final answer) tuple and returns a scalar - usually a probability that the final answer is correct. It is what you get if you train a reward model the standard RLHF way on (problem, solution, correct/wrong) data.
A Process Reward Model (PRM) takes a (prompt, partial chain up to step k) and returns a per-step score - the probability that the reasoning so far is on the right track. Training labels are step-level: a human (or a strong model) marks each step as correct, neutral, or incorrect.
The difference at inference time:
# ORM-guided best-of-N
candidates = [sample_full_solution() for _ in range(N)]
scores = [orm(prompt, c) for c in candidates]
return candidates[argmax(scores)]
# PRM-guided beam search
beam = [empty_chain]
for step in range(max_steps):
expansions = [c.extend(sample_step(c)) for c in beam for _ in range(k)]
scores = [prm(prompt, e) for e in expansions]
beam = top_b(expansions, scores) # keep best-b partial chains
return best(beam)
ORMs prune after the fact. PRMs prune during search - cutting off doomed branches early and reallocating compute to live ones.
Let's Verify Step by Step
The canonical PRM paper is Lightman, Kosaraju, Burda et al, OpenAI 2023: "Let's Verify Step by Step" (arXiv 2305.20050). Setup: GSM8K-style maths, then MATH. Two reward models trained on the same prompts:
- An ORM on (solution, final-answer-correct) labels.
- A PRM on PRM800K - 800k step-level human annotations of GPT-4-generated solutions.
Result: best-of-N with the PRM solved 78% of a representative MATH test subset, materially beating the ORM at the same N. The gap widened with N: PRMs scale better with sampling because they prune wrong-direction chains earlier.
A second finding from the same line of work: PRMs can be trained on automatically-labelled data (Math-Shepherd, OmegaPRM) by Monte-Carlo-rolling-out each step and labelling it correct if it leads to a correct answer often enough. This is the bridge from "PRMs are great but labelling is impossible at scale" to "PRMs are great and we can synthesise the labels".
RLVR: skip the reward model entirely
For domains where correctness is programmatically checkable - maths with a known answer, code with unit tests, formal proofs - you do not need a learned reward model at all. You can run RL against a deterministic verifier.
That is the core idea of Reinforcement Learning with Verifiable Rewards (RLVR), formalised in Ai2's Tülu 3 paper (Lambert et al, November 2024). The recipe:
- Curate prompts whose answers are checkable by a function (MATH, GSM8K, BBH, IFEval-style constraints).
- For each prompt, sample completions from the policy.
- Reward = 1 if the verifier accepts the answer, 0 otherwise. No PRM, no preference model.
- Run policy optimisation (Tülu 3 used PPO; DeepSeek-R1 used GRPO, a critic-free variant).
The signal is sparse but unhackable in the usual RLHF sense - the model cannot win by being verbose, flattering, or fluent. It wins only by being right. That property is what makes RLVR cheap enough to scale.
DeepSeek-R1 (January 2025) pushed RLVR further: they ran RL directly on a base model with no SFT cold-start and watched reasoning behaviours - reflection, backtracking, "wait, let me try again" - emerge from the verifier signal alone. That run produced R1-Zero. The released R1 added a small SFT cold-start for readability, but the load-bearing training signal was still RLVR.
Why process rewards help on multi-step problems
On a 30-step proof, an ORM tells you the proof was wrong. A PRM (or a verifier with step-level checkpoints) tells you step 17 was wrong. Three concrete benefits:
- Earlier termination. Search drops branches before burning tokens on them.
- Better credit assignment in RL. Gradient flows to the step that actually mattered, not smeared across the whole chain.
- Calibration. The model learns its own uncertainty step-wise, which transfers to better self-correction at inference.
The catch is that for most real-world tasks neither approach works cleanly. "Write a good design doc" has no verifier and no obvious step-level ground truth. RLVR shines on maths and code; PRMs shine where you can afford the labels.
The labelling cost problem
PRM800K cost OpenAI an estimated mid-six figures in labeller time. Scaling that to every domain a frontier model is supposed to reason about is not feasible. The field's responses:
| Approach | What it does | Trade-off |
|---|---|---|
| Human step labels (Let's Verify) | Hire experts to annotate each step | Highest quality, lowest scale |
| MC roll-out auto-labels (Math-Shepherd) | Label a step correct if MC rollouts from it succeed often | Cheap, noisy, biased toward easy steps |
| LLM-as-judge step labels | Ask a stronger model to grade each step | Scales, inherits judge biases |
| RLVR with programmatic verifier | Drop step labels entirely; reward only on checkable outcomes | Sparse signal, only works in verifiable domains |
| Hybrid (cold-start SFT + RLVR + occasional PRM) | DeepSeek-R1 style | Best current results, most complex pipeline |
Where it falls down
- Verifier-induced specialisation. A model trained heavily with RLVR on maths and code can regress on conversational tasks. Tülu 3 and R1 both report this and counter it with mixed-domain training.
- Reward hacking the verifier. Models discover that printing the expected answer format without the reasoning fools weak checkers. Robust verifiers and unit-test suites with adversarial cases mitigate but do not eliminate.
- PRMs as adversarial surfaces. A PRM that scores intermediate steps becomes an optimisation target. Models can learn step shapes the PRM rewards even when the reasoning is hollow. This is the long-chain analogue of length-bias in RLHF.
Further reading
5 flashcards for this concept
Click a card to reveal the answer.