The reasoning-model bubble: when test-time compute stops paying

The reasoning-model wave is the first capability jump in this cycle whose economics actively work against most product use cases. o1, o3, R1 and Claude extended thinking are genuine breakthroughs on a narrow band of tasks (verifiable math, competition coding, multi-step synthesis with a checkable answer) and a tax on almost everything else. The hero benchmarks - AIME, FrontierMath, ARC-AGI - reward exactly the shape of problem that pays back for thirty seconds of hidden token spend. Your support chat, your search box, your agent loop, and your in-product copilot reward exactly the opposite shape. The product teams that miss this distinction in 2026 will spend the year wondering why their unit economics broke and why their NPS scores fell.

This is not an anti-reasoning essay. The capability is real, the underlying RL research is the most important post-training story since RLHF, and there are product categories where these models are the only honest answer. The argument is narrower: the reasoning frame has been over-applied, the latency and cost math is being waved away, and a meaningful fraction of integrations shipping right now would have shipped better against a fast model with a tighter prompt.

Where the capability actually lives

The mechanism is well-understood at this point. Snell et al. (2024) showed that compute-optimal test-time scaling could improve efficiency more than 4x over best-of-N, and in a FLOPs-matched comparison let a smaller base model outperform a 14x larger one on problems where the smaller model already had a reasonable baseline success rate. That last clause is the load-bearing one. Test-time compute amplifies signal that already exists; it does not manufacture signal from noise. DeepSeek-R1 reproduced the recipe in the open with pure RL on verifiable rewards (math, code, STEM) and shipped weights with an inference price (\(0.55/\)2.19 per million input/output tokens) that made it impossible to ignore. OpenAI's o3 sits at \(2/\)8 per million. Claude 3.7 Sonnet, with or without extended thinking, is \(3/\)15.

The economic property to internalise: reasoning tokens are billed as output tokens, they are invisible in the response, and the model decides how many to generate. A single o3 query on a non-trivial prompt routinely consumes 5,000-25,000 hidden reasoning tokens before the visible answer begins. At $8 per million output, that is between four cents and twenty cents per call before the user has seen a single character. Anthropic is explicit that "accuracy on math questions improves logarithmically with the number of thinking tokens". Logarithmic. You double the spend, you nudge accuracy by a constant. This is the curve product managers need to be staring at.

The underlying mechanism is process-supervised RL, traced back to Lightman et al's "Let's Verify Step by Step" (2023) which showed that step-level reward modelling beat outcome-only supervision and pushed MATH-subset performance to 78%. That is the lineage. The recipe works. The question this essay asks is not whether the recipe works - it does - but where the recipe earns its keep in production.

The latency wall that benchmarks hide

Hero benchmarks are scored offline. Products run online and the user is timing them. The numbers from Artificial Analysis as of May 2026 tell a story that the leaderboards do not.

Conditions: TTFT is median p50 on a single-turn 500-token completion as reported by Artificial Analysis; E2E ranges are observed in production traffic on prompts that trigger non-trivial reasoning. Llama-70B figures are from Groq's published rates.

The o3 TTFT alone is roughly an order of magnitude worse than GPT-4o's. End-to-end, a "thinking" response with a substantive answer routinely lands in the 30-90 second range. Simon Willison's pelican-SVG test of GPT-5-Pro famously took 6 minutes 8 seconds for a single drawing. That is the upper bound of the regime, but the median is bad enough that it breaks every chat UX heuristic we have from the last two decades of web. The Nielsen "1 second feels instant, 10 seconds is the limit before users disengage" research did not contemplate watching a model think for the length of a TV commercial break.

The product-level lesson: users do not perceive "the model is thinking carefully" as a feature. They perceive a frozen screen. The exposed reasoning trace - the thing the labs are proudest of - reads as either reassuring or condescending depending on the user, and converts to neither retention nor revenue in any A/B test I have seen run. The benchmark says +12 points on FrontierMath. The session metrics say -18% on engaged sessions. Both are true. Only one of them is on your dashboard.

Where the extra accuracy compounds to revenue

Three product categories where reasoning models are unambiguously the right call.

• High-stakes single-shot tasks where the user is already waiting. Code review on a PR. Deep research reports. Math tutoring with a checkable answer. Legal contract analysis. The user has already accepted that the work takes minutes, the cost of a wrong answer is meaningful, and the cost of the right answer absorbs a fifty-cent inference bill without flinching. Willison's framing of o1 as a "report generator, not a chat model" is the cleanest mental model I have seen for this category.
• Verifiable-reward workflows. Anywhere your task has a programmatic check at the end - unit tests pass, the SQL returns the right row, the proof type-checks - reasoning models eat fast models alive. This is the category Nathan Lambert keeps pointing at: RL on verifiable rewards is where the new training paradigm actually works. SWE-bench Verified resolve rates climbed past 70% in 2025 precisely because the loop has a real signal.
• Tasks where a human is going to verify the output anyway. A radiologist double-checking a draft read. An analyst sanity-checking a model's table extraction. The reasoning model takes 60 seconds to produce a better candidate; the human spends 2 minutes verifying instead of 8 minutes producing. The asymmetric speedup is real.

Where it does not compound, and quietly destroys margin

Five product categories where the reasoning frame ships a worse product than a fast model with a tighter prompt.

• Conversational UX. Any feature where the user expects a back-and-forth at human-speech latency. Customer support chat. Onboarding assistants. In-product help. The cost of a 30-second pause is not "slightly worse" - it is a different product. Users abandon. Sessions end. The accuracy delta on these tasks (typically 1-3 percentage points on a well-tuned eval) does not move retention. The latency delta (5-25x) does.
• Simple lookups and classification. Routing a ticket. Extracting fields from an email. Classifying an intent. These are tasks where a fine-tuned classifier or a fast model with a structured-output schema scores 95%+ and a reasoning model scores 96-97% for 20x the cost and 30x the latency. Lambert pointed out that o1 will burn 225 tokens on a query a normal model solves in 10-12. That is a 20x tax on the simplest path through your product.
• Agentic workflows with many fast iterations. A coding agent that runs 30 tool calls per task does not want each call to take 45 seconds. Cognition's own post on Devin is implicitly about how fast iteration beats slow deliberation in long loops. The right architecture is usually a fast model in the loop and a reasoning model called once at a planning or verification checkpoint - not a reasoning model end-to-end.
• High-volume B2C surfaces with thin per-call margin. A consumer app serving a million chat turns a day at o3 economics is paying \(40-200k a day in inference. The same workload on a fast model is \)2-10k. There is no consumer pricing model that absorbs the difference.
• Streaming or progressive UX. Reasoning models cannot stream a useful answer because the thinking has to finish before the visible response begins. Every UI pattern built around "show the answer as it generates" - copilots, autocomplete, live transcription - is structurally incompatible with the reasoning frame.

A decision table for the product lead

When a feature lands on the roadmap, run it through this grid before defaulting to o3.

The single most useful exercise: take any feature where you have defaulted to a reasoning model and ask, honestly, which column you are actually in. The number of times the answer flips on closer inspection is the size of the bubble.

The cost math, run forward

A worked example to make this concrete. Imagine a B2B support chatbot serving 50,000 sessions per day, 8 turns per session, 800 input tokens and 300 output tokens per turn.

• GPT-4o: 400,000 turns/day at (800 input + 300 output) per turn, costed at \(2.50/M input and \)10/M output, lands around $2,000/day. P50 latency under 3 seconds.
• o3 with reasoning: assume the same visible answer plus 5,000 hidden reasoning tokens per turn. 400,000 turns at (800 input + 5,300 output) per turn, costed at \(2/M input and \)8/M output, lands around $17,600/day. P50 latency 30-60 seconds.
• Cost delta: ~9x. Latency delta: ~12-20x. Accuracy delta on support tasks: typically 1-3 percentage points if you have a decent retrieval stack, and the reasoning model often does worse on tone and brevity.

The product question is not "should we use the most capable model." It is "does the extra 1-3 points of accuracy justify a 9x cost increase, a 15x latency increase, and a measurable drop in conversational quality?" For almost every B2B support use case I have seen, the answer is no. For a tax assistant doing a one-shot multi-jurisdiction filing, the answer is yes. The shape of the task is what determines this, not the prestige of the model.

What the contrarians have right, and where the bulls do too

The bull case for reasoning models is not crazy. Epoch AI's work on inference scaling shows that 2 orders of magnitude of training compute can be traded for 3-4 orders of magnitude of inference compute at small scales, falling to roughly a 4:6 ratio near the capability ceiling. That trade favours inference scaling for any team that cannot run a $500M training cluster - which is almost everyone. The case for reasoning models as a research direction is overwhelming.

The case for them as a product default is much weaker, and the gap between "this is technically impressive" and "this should be in production" is where the bubble lives. Lambert has been clear that o1 shipped as "far from a coherent product" - a prototype that vendors then encouraged enterprises to integrate as if it were a finished product. The capability research and the productisation are different problems. Conflating them is what is going to blow up budgets in Q3.

The contrarians are right that most chat-shaped products will not improve and will likely degrade if you swap in a reasoning model. The bulls are right that math tutors, code review tools, deep-research products, agent planning checkpoints, and any workflow with a verifiable reward will quietly become 2-5x more useful over the next 18 months because of this capability. Both are simultaneously true.

How to integrate reasoning models without overpaying

A practical checklist for the engineering lead who has been told to "add reasoning."

1. Route by task structure, not by user request. A small router model (think Llama-3-8B or Haiku) classifies the incoming task as math-y / verifiable / open-ended / chat. Only the first two categories get routed to a reasoning model. The rest go to a fast model. Most of your traffic will be in the rest.
2. Cap the thinking budget explicitly. Both Claude and o3 expose this. Pick a budget that matches the task class. Set 1k tokens for routine work, 8k for hard work, 32k only for the cases where the cost is justified. The default is wildly over-provisioned.
3. Use the reasoning model as a verifier, not a generator. A fast model produces N candidates. The reasoning model picks the best one or critiques the top candidate. You get most of the accuracy benefit at a fraction of the latency and cost, because the verification call is shorter than the generation call.
4. Cache aggressively. A reasoning model's output for a deterministic question is worth caching for days, not minutes. The marginal cost of a cache hit is zero. The marginal cost of a cache miss is large. Build the cache layer before you ship the feature.
5. Run two evals, not one. A task-accuracy eval (does the answer match the golden set) and a latency-distribution eval (p50, p95, p99 wall-clock from request to last token). A reasoning model that wins the first and loses the second is not a win. Both need to clear bar.
6. Instrument the visible-reasoning UX explicitly. If you show the thinking trace to users, A/B test whether it improves any business metric. In our experience it improves perceived trust on research tasks and erodes it on chat tasks. Do not assume - measure.
7. Plan the fallback. When the reasoning model is slow, expensive, or down, what does the product do? A graceful degradation path to a fast model with a "less detailed" badge keeps the SLA. No fallback path is a vulnerability.
8. Re-evaluate every quarter. The fast-model accuracy ceiling is climbing roughly twice as fast as the reasoning-model ceiling, because reasoning models are downstream of base-model quality. The reasoning-model premium is shrinking in real time. A decision you made in Q1 2026 may be wrong by Q3 2026.

The reasoning-model wave is the first capability that asks product teams to think about hidden tokens, multi-modal latency budgets, and progressive UX incompatibility as first-class design constraints. Most teams will get this wrong on the first pass. The ones who get it right will treat reasoning as a routing decision, not a default - and they will ship products that feel both smarter and faster than their competitors, because they will only pay for thinking when thinking earns its keep.

The hero benchmarks were the marketing. The latency tail is the product.

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.