Most teams treat prompt caching as a billing hack. The deeper consequence is that you can now ship product patterns - persistent agents, large-context tools, system-prompt-heavy UX - that were latency-prohibitive eighteen months ago.
DeepSeek shipped a 1.6T-parameter model in April 2026 with native Ascend inference support while OpenAI committed $500B to Stargate. Both can be right, but only one of them is a moat.
Training a frontier model is a one-time expense measured in millions. Serving it to users is a perpetual expense measured in billions of tokens per day. For most organisations, inference is already the larger number, and the gap …
o3, R1 and Claude extended thinking are a real capability shift on a narrow slice of tasks. They are also being shoved into product surfaces that punish every property reasoning models exhibit - and the bill is starting to arrive.