Inference-Time Scaling Laws and Thinking Budgets

Pretraining scaling laws relate loss to parameters and data. Inference-time scaling laws relate accuracy to the compute you spend after the weights are frozen, and they have a different shape: roughly logarithmic in tokens, with a ceiling, and with an optimal strategy that changes by question.

The practical consequence is that "think longer" is a hyperparameter with a cost curve, not a setting to max out.

The two axes of test-time compute

Sequential. One chain that goes deeper: longer CoT, revision, self-refinement. Each token conditions on all prior reasoning, so the model can build on partial progress. It fails when an early error poisons everything downstream, because there is no mechanism to abandon a bad path.

Parallel. Many independent chains that are then aggregated: best-of-\(N\) against a verifier, self-consistency by majority vote, or beam and lookahead search guided by a process reward model. Independence is the point; one bad chain does not contaminate the others. It fails when the model's errors are systematic, because \(N\) samples reproduce the same mistake \(N\) times and the majority vote confidently returns it.

Snell and colleagues showed the choice between them is not fixed. On easier questions where the base model's first attempt is nearly right, sequential revision wins. On harder questions where the model needs to find a different approach entirely, parallel search wins. Selecting adaptively per question, what they call compute-optimal scaling, improved test-time compute efficiency by more than 4x over a best-of-\(N\) baseline (Snell et al., 2024, arXiv:2408.03314).

The trade against pretraining

The same work asked the question that makes this a scaling law rather than a trick: given a fixed FLOP budget, is it better spent on a bigger model or on more inference compute for a smaller one?

The answer depends on the difficulty of the question and on how many times you will run inference. For easy and medium questions at modest inference volume, extra test-time compute on a smaller model can substitute for a substantially larger one. For the hardest questions, and for high-volume deployments where the inference cost is paid on every request forever, pretraining compute wins. A model trained once and served a billion times amortises training cost to nothing, which is the regime most production systems are in.

What a thinking budget controls

Reasoning models expose a budget as a cap on reasoning tokens. It is worth being precise about what it does.

It is an upper bound, not a target: the model may stop early on an easy question, so cost tracks difficulty rather than the cap. It applies to hidden reasoning tokens, which are billed as output but usually not returned, so budget and visible answer length are unrelated. And it truncates rather than compresses: hitting the cap mid-chain forces an answer from an incomplete state, which is often worse than a shorter chain would have been if the model had known its budget.

Budget forcing exploits this deliberately, appending a "Wait" token to make a model continue past its natural stopping point, and shows accuracy improving with forced length up to a point (Muennighoff et al., 2025, s1: Simple Test-Time Scaling, arXiv:2501.19393).

When it breaks

Overthinking. Beyond a task-dependent point, longer chains reduce accuracy. The model talks itself out of a correct early answer, a pattern most pronounced on easy questions where the first instinct was right.

Verifier quality caps parallel scaling. Best-of-\(N\) is bounded by the ability to pick the winner. With a weak verifier, increasing \(N\) eventually hurts, because more samples give the verifier more chances to select a plausible wrong answer. Majority voting has the same failure when errors are correlated.

Latency does not amortise. Parallel sampling can be batched, so it costs money but not wall-clock. Sequential reasoning costs wall-clock directly: 8,000 reasoning tokens at 40 ms each is more than five minutes before the user sees anything. For interactive products that constraint binds long before the cost one does.

Benchmark gains overstate product gains. Test-time compute helps most on verifiable, well-posed, hard problems. On open-ended writing or ambiguous retrieval it does much less, and the published scaling curves come almost entirely from maths and code.