Inference & Serving

Latency, throughput, KV-cache economics, quantisation and the real cost of tokens.

All Model Architecture13 Training & Alignment8 Inference & Serving11 Agents & Orchestration10 Reasoning & Evaluation7 Safety, Security & Governance3 Platforms & Practice11

Inference & Serving 21 min

Cache-Augmented Generation: When Preloaded KV-Caches Replace Retrieval Pipelines

Retrieval-augmented generation fetches documents at query time, scores them, and hopes the retriever got it right. Cache-Augmented Generation sidesteps the entire pipeline by preloading knowledge into the model's KV-cache before …

kv-cache cache-augmented-generation transformer-inference rag-alternative ∑ ◫

Inference & Serving 21 min

Diffusion Language Models: Writing Text by Denoising, Not Predicting the Next Token

Autoregressive models write left to right, one token at a time. Diffusion language models reveal a whole sequence at once and sharpen it over a handful of steps. That single change rewrites the latency math, and in 2025 it stoppe…

diffusion-models llm parallel-decoding generative-ai ∑ ◫

Inference & Serving 21 min

FlashAttention and the Memory Wall: Why Attention Was Never Compute-Bound

An A100 can execute 312 trillion half-precision operations per second but can only pull about 2 terabytes from memory in that same second. FlashAttention made attention fast not by computing less, but by refusing to touch memory.

flashattention gpu attention transformers ∑ ◫

Inference & Serving 21 min

Four Bits Per Weight: How Low-Precision Quantization Stopped Hurting LLMs

A 70B model in FP16 needs 140 GB of memory it spends most of its time waiting to read. Dropping each weight to four bits cuts that to 35 GB, and for years that cut also broke the model. Here is what changed.

quantization inference gptq awq ∑ ◫

Inference & Serving 24 min

Goodput, Not Throughput: The Metric That Decides Whether Your LLM Deployment Works

Two servers run the same model on the same GPUs. One reports 4,200 tokens per second and is unusable; the other reports 2,600 and feels instant. Throughput is a property of the server, latency is a property of the request, and th…

inference serving latency goodput ∑ ◫

Inference & Serving 21 min

PagedAttention and Continuous Batching: How vLLM Stopped Wasting Your GPU

A GPU loaded with a 13B model can have most of its KV-cache memory sitting idle while requests queue for capacity. PagedAttention and continuous batching reclaim that memory, and the throughput follows.

inference vllm kv-cache gpu-serving ∑ ◫

Inference & Serving 4 min

Prompt caching is not just a cost optimisation - it changes what apps you can build

Most teams treat prompt caching as a billing hack. The deeper consequence is that you can now ship product patterns - persistent agents, large-context tools, system-prompt-heavy UX - that were latency-prohibitive eighteen months ago.

caching inference latency economics

Inference & Serving 21 min

Sleep-Time Compute: Thinking About the Context Before the Question Arrives

Test-time compute makes models smarter but slower and pricier. Sleep-time compute moves the thinking to before the user asks, cutting the compute needed for the same accuracy by roughly 5x and the cost per query by 2.5x when cont…

inference test-time-compute agents reasoning ∑ ◫

Inference & Serving 23 min

Speculative Decoding: How a Small Draft Model Makes Large Language Models Think Faster

Autoregressive decoding forces even the largest language models to emit one token at a time, bottlenecked by memory bandwidth rather than compute. Speculative decoding breaks this sequential constraint by letting a lightweight dr…

speculative-decoding llm-inference parallel-decoding draft-model ∑ ◫

Inference & Serving 25 min

Structured Generation and Constrained Decoding: Making LLMs Predictable

Language models generate text one token at a time by sampling from a probability distribution over their entire vocabulary. Constrained decoding intervenes at that sampling step, masking out every token that would violate a targe…

structured-output constrained-decoding JSON-mode FSM ∑ ◫

Inference & Serving 21 min

The Hidden Cost of AI: Why Inference Is the New Cloud Bill

Training a frontier model is a one-time expense measured in millions. Serving it to users is a perpetual expense measured in billions of tokens per day. For most organisations, inference is already the larger number, and the gap …

inference economics optimization finops ∑ ◫

Browse by topic

Inference & Serving

Cache-Augmented Generation: When Preloaded KV-Caches Replace Retrieval Pipelines

Diffusion Language Models: Writing Text by Denoising, Not Predicting the Next Token

FlashAttention and the Memory Wall: Why Attention Was Never Compute-Bound

Four Bits Per Weight: How Low-Precision Quantization Stopped Hurting LLMs

Goodput, Not Throughput: The Metric That Decides Whether Your LLM Deployment Works

PagedAttention and Continuous Batching: How vLLM Stopped Wasting Your GPU

Prompt caching is not just a cost optimisation - it changes what apps you can build

Sleep-Time Compute: Thinking About the Context Before the Question Arrives

Speculative Decoding: How a Small Draft Model Makes Large Language Models Think Faster

Structured Generation and Constrained Decoding: Making LLMs Predictable

The Hidden Cost of AI: Why Inference Is the New Cloud Bill

Tags