Tagged “inference”

Inference & Serving 4 min

Prompt caching is not just a cost optimisation - it changes what apps you can build

Most teams treat prompt caching as a billing hack. The deeper consequence is that you can now ship product patterns - persistent agents, large-context tools, system-prompt-heavy UX - that were latency-prohibitive eighteen months ago.

caching inference latency economics

Inference & Serving 21 min

Sleep-Time Compute: Thinking About the Context Before the Question Arrives

Test-time compute makes models smarter but slower and pricier. Sleep-time compute moves the thinking to before the user asks, cutting the compute needed for the same accuracy by roughly 5x and the cost per query by 2.5x when cont…

inference test-time-compute agents reasoning ∑ ◫

Inference & Serving 25 min

Structured Generation and Constrained Decoding: Making LLMs Predictable

Language models generate text one token at a time by sampling from a probability distribution over their entire vocabulary. Constrained decoding intervenes at that sampling step, masking out every token that would violate a targe…

structured-output constrained-decoding JSON-mode FSM ∑ ◫

Reasoning & Evaluation 18 min

Test-Time Compute: How Reasoning Models Buy Intelligence by the Token

A 7B model that thinks for thirty seconds can outscore a 70B model that answers instantly. Test-time compute is the lever that makes this possible, and it is quietly rewriting the economics of model design.

reasoning test-time-compute inference reinforcement-learning ∑ ◫

Inference & Serving 21 min

The Hidden Cost of AI: Why Inference Is the New Cloud Bill

Training a frontier model is a one-time expense measured in millions. Serving it to users is a perpetual expense measured in billions of tokens per day. For most organisations, inference is already the larger number, and the gap …

inference economics optimization finops ∑ ◫

Inference & Serving 27 min

The Mode Is Not the Answer: How Decoding Stopped Being a Search Problem

Run exact search over a trained translation model and, for more than half of a standard test set, the highest-scoring output is the empty string. Beam search works only because it fails to find what the model actually prefers. Th…

decoding generation sampling mbr ∑ ◫

Safety, Security & Governance 21 min

The Moderation Tax: How Guardrail Classifiers Trade Latency for Coverage

A guardrail is a classifier sandwich wrapped around your model, and every layer you add buys coverage with latency and false refusals. Here is how the layer actually works, what it costs, and where it breaks.

llm-safety guardrails content-moderation jailbreaks ∑ ◫

Reasoning & Evaluation 13 min

The reasoning-model bubble: when test-time compute stops paying

o3, R1 and Claude extended thinking are a real capability shift on a narrow slice of tasks. They are also being shoved into product surfaces that punish every property reasoning models exhibit - and the bill is starting to arrive.

reasoning test-time-compute economics evaluation ∑

Model Architecture 18 min

Trainable Sparse Attention: When the Model Learns What to Skip

For years, sparse attention was a bolt-on applied after training to shrink the KV cache. NSA, MoBA, and DeepSeek's DSA flip that order, training the sparsity end to end and reporting up to 11x faster decoding without losing accuracy.

attention long-context sparse-attention transformers ∑ ◫

Inference & Serving 24 min

Your Answer Depends on Who Else Is in the Batch

A thousand identical requests to the same model at temperature zero produced eighty different completions. The cause is not random seeds, not sampling, and not GPU atomics. It is that reduction kernels change their summation orde…

inference determinism numerics kernels ∑ ◫