Latency, throughput, KV-cache economics, quantisation and the real cost of tokens.
Test-time compute makes models smarter but slower and pricier. Sleep-time compute moves the thinking to before the user asks, cutting the compute needed for the same accuracy by roughly 5x and the cost per query by 2.5x when cont…
Autoregressive decoding forces even the largest language models to emit one token at a time, bottlenecked by memory bandwidth rather than compute. Speculative decoding breaks this sequential constraint by letting a lightweight dr…
Language models generate text one token at a time by sampling from a probability distribution over their entire vocabulary. Constrained decoding intervenes at that sampling step, masking out every token that would violate a targe…
Training a frontier model is a one-time expense measured in millions. Serving it to users is a perpetual expense measured in billions of tokens per day. For most organisations, inference is already the larger number, and the gap …
Run exact search over a trained translation model and, for more than half of a standard test set, the highest-scoring output is the empty string. Beam search works only because it fails to find what the model actually prefers. Th…
A thousand identical requests to the same model at temperature zero produced eighty different completions. The cause is not random seeds, not sampling, and not GPU atomics. It is that reduction kernels change their summation orde…