Inference & Serving

Latency, throughput, KV-cache economics, quantisation and the real cost of tokens.

All Model Architecture19 Training & Alignment22 Inference & Serving18 Agents & Orchestration11 Reasoning & Evaluation26 Safety, Security & Governance7 Platforms & Practice20

Inference & Serving 29 min

Arithmetic Intensity: Why Your GPU Is Idle 99% of the Time

An H100 advertises 989 teraflops. Generating one token from an 8B model uses roughly 0.3% of that. The gap is not a bug in your code or a missing compiler flag; it is a single ratio, FLOPs per byte moved, and almost every perform…

neural-plumbing gpu performance kernels ∑ ◫

Inference & Serving 21 min

Cache-Augmented Generation: When Preloaded KV-Caches Replace Retrieval Pipelines

Retrieval-augmented generation fetches documents at query time, scores them, and hopes the retriever got it right. Cache-Augmented Generation sidesteps the entire pipeline by preloading knowledge into the model's KV-cache before …

kv-cache cache-augmented-generation transformer-inference rag-alternative ∑ ◫

Inference & Serving 24 min

Designing for a Collaborator That Is Sometimes Wrong: The Evidence Behind Human-AI Interaction Design

In a 2025 randomized trial, experienced developers using AI tools took 19% longer to finish their tasks while believing they had been 20% faster. Twenty-six years of human-AI interaction research explain the gap: an assistant's v…

interaction-design-for-ai product verification calibration ∑ ◫

Inference & Serving 21 min

Diffusion Language Models: Writing Text by Denoising, Not Predicting the Next Token

Autoregressive models write left to right, one token at a time. Diffusion language models reveal a whole sequence at once and sharpen it over a handful of steps. That single change rewrites the latency math, and in 2025 it stoppe…

diffusion-models llm parallel-decoding generative-ai ∑ ◫

Inference & Serving 27 min

Everything Is Lossy Compression: A Rate-Distortion View of Quantisation, KV Caches, and Distillation

Weight quantisation, KV cache eviction, prompt compression and distillation are treated as four separate engineering disciplines with four separate literatures. They are one problem: choosing a point on a rate-distortion curve. S…

information-theory quantisation kv-cache distillation ∑ ◫

Inference & Serving 21 min

FlashAttention and the Memory Wall: Why Attention Was Never Compute-Bound

An A100 can execute 312 trillion half-precision operations per second but can only pull about 2 terabytes from memory in that same second. FlashAttention made attention fast not by computing less, but by refusing to touch memory.

flashattention gpu attention transformers ∑ ◫

Inference & Serving 21 min

Four Bits Per Weight: How Low-Precision Quantization Stopped Hurting LLMs

A 70B model in FP16 needs 140 GB of memory it spends most of its time waiting to read. Dropping each weight to four bits cuts that to 35 GB, and for years that cut also broke the model. Here is what changed.

quantization inference gptq awq ∑ ◫

Inference & Serving 24 min

Goodput, Not Throughput: The Metric That Decides Whether Your LLM Deployment Works

Two servers run the same model on the same GPUs. One reports 4,200 tokens per second and is unusable; the other reports 2,600 and feels instant. Throughput is a property of the server, latency is a property of the request, and th…

inference serving latency goodput ∑ ◫

Inference & Serving 25 min

How Many Tokens Is an Image? The Resolution Policy That Decides Your VLM Bill

The same 1024x1024 screenshot costs 576 visual tokens in one model and 2,880 in another, and the cheap one cannot read the text. The rule that turns pixels into tokens is the least examined hyperparameter in the multimodal stack,…

vision-multimodal vlm tokenisation inference ∑ ◫

Inference & Serving 21 min

PagedAttention and Continuous Batching: How vLLM Stopped Wasting Your GPU

A GPU loaded with a 13B model can have most of its KV-cache memory sitting idle while requests queue for capacity. PagedAttention and continuous batching reclaim that memory, and the throughput follows.

inference vllm kv-cache gpu-serving ∑ ◫

Inference & Serving 4 min

Prompt caching is not just a cost optimisation - it changes what apps you can build

Most teams treat prompt caching as a billing hack. The deeper consequence is that you can now ship product patterns - persistent agents, large-context tools, system-prompt-heavy UX - that were latency-prohibitive eighteen months ago.

caching inference latency economics

Inference & Serving 24 min

Running Language Models on a Phone: Memory Bandwidth, NPUs, and the Few-Billion-Parameter Ceiling

Phones ship NPUs rated in trillions of operations per second, yet the speed at which a reply appears is set by how fast LPDDR memory can hand a couple of gigabytes of weights to the processor, over and over. This post derives the…

on-device-and-edge-ai on-device-ai llm-inference memory-bandwidth ∑ ◫

Browse by topic

Inference & Serving

Arithmetic Intensity: Why Your GPU Is Idle 99% of the Time

Cache-Augmented Generation: When Preloaded KV-Caches Replace Retrieval Pipelines

Designing for a Collaborator That Is Sometimes Wrong: The Evidence Behind Human-AI Interaction Design

Diffusion Language Models: Writing Text by Denoising, Not Predicting the Next Token

Everything Is Lossy Compression: A Rate-Distortion View of Quantisation, KV Caches, and Distillation

FlashAttention and the Memory Wall: Why Attention Was Never Compute-Bound

Four Bits Per Weight: How Low-Precision Quantization Stopped Hurting LLMs

Goodput, Not Throughput: The Metric That Decides Whether Your LLM Deployment Works

How Many Tokens Is an Image? The Resolution Policy That Decides Your VLM Bill

PagedAttention and Continuous Batching: How vLLM Stopped Wasting Your GPU

Prompt caching is not just a cost optimisation - it changes what apps you can build

Running Language Models on a Phone: Memory Bandwidth, NPUs, and the Few-Billion-Parameter Ceiling

Tags