Tagged “inference”

Training & Alignment 21 min

Borrowed Intelligence: How Knowledge Distillation Builds Small Language Models That Punch Above Their Weight

A 2-billion-parameter model that trades blows with one ten times its size is not an accident of architecture. It is the product of a teacher pouring its full probability distribution into a student, token by token.

distillation small-language-models model-compression on-device-ai ∑ ◫

Inference & Serving 21 min

Diffusion Language Models: Writing Text by Denoising, Not Predicting the Next Token

Autoregressive models write left to right, one token at a time. Diffusion language models reveal a whole sequence at once and sharpen it over a handful of steps. That single change rewrites the latency math, and in 2025 it stoppe…

diffusion-models llm parallel-decoding generative-ai ∑ ◫

Model Architecture 20 min

Do Transformers Need Three Projections? Rethinking Q, K, and V

Every attention head learns three weight matrices for query, key, and value. A 2026 study trained models up to 1.2B parameters to ask which of them are actually load-bearing, and found that keys and values can share one projectio…

transformers attention kv-cache inference ∑ ◫

Inference & Serving 21 min

FlashAttention and the Memory Wall: Why Attention Was Never Compute-Bound

An A100 can execute 312 trillion half-precision operations per second but can only pull about 2 terabytes from memory in that same second. FlashAttention made attention fast not by computing less, but by refusing to touch memory.

flashattention gpu attention transformers ∑ ◫

Inference & Serving 21 min

Four Bits Per Weight: How Low-Precision Quantization Stopped Hurting LLMs

A 70B model in FP16 needs 140 GB of memory it spends most of its time waiting to read. Dropping each weight to four bits cuts that to 35 GB, and for years that cut also broke the model. Here is what changed.

quantization inference gptq awq ∑ ◫

Inference & Serving 24 min

Goodput, Not Throughput: The Metric That Decides Whether Your LLM Deployment Works

Two servers run the same model on the same GPUs. One reports 4,200 tokens per second and is unusable; the other reports 2,600 and feels instant. Throughput is a property of the server, latency is a property of the request, and th…

inference serving latency goodput ∑ ◫

Model Architecture 20 min

Half Mamba, Half Attention: Why Hybrid State-Space Models Took Over

Pure Mamba was supposed to replace attention. Instead the most efficient open models in 2025 are roughly seven-eighths Mamba and one-eighth attention. The reason is the KV cache, and what each layer can and cannot remember.

state-space-models mamba transformers long-context ∑ ◫

Model Architecture 25 min

Mixture of Experts: How Frontier LLMs Achieve Scale Without Proportional Cost

Every frontier LLM released since 2023 uses some form of Mixture of Experts. The idea is deceptively simple: build a model with hundreds of billions of parameters but activate only a fraction for each token. The result is dense-m…

moe architecture scaling inference ∑ ◫

Inference & Serving 21 min

PagedAttention and Continuous Batching: How vLLM Stopped Wasting Your GPU

A GPU loaded with a 13B model can have most of its KV-cache memory sitting idle while requests queue for capacity. PagedAttention and continuous batching reclaim that memory, and the throughput follows.

inference vllm kv-cache gpu-serving ∑ ◫

Inference & Serving 4 min

Prompt caching is not just a cost optimisation - it changes what apps you can build

Most teams treat prompt caching as a billing hack. The deeper consequence is that you can now ship product patterns - persistent agents, large-context tools, system-prompt-heavy UX - that were latency-prohibitive eighteen months ago.

caching inference latency economics

Inference & Serving 21 min

Sleep-Time Compute: Thinking About the Context Before the Question Arrives

Test-time compute makes models smarter but slower and pricier. Sleep-time compute moves the thinking to before the user asks, cutting the compute needed for the same accuracy by roughly 5x and the cost per query by 2.5x when cont…

inference test-time-compute agents reasoning ∑ ◫

Inference & Serving 25 min

Structured Generation and Constrained Decoding: Making LLMs Predictable

Language models generate text one token at a time by sampling from a probability distribution over their entire vocabulary. Constrained decoding intervenes at that sampling step, masking out every token that would violate a targe…

structured-output constrained-decoding JSON-mode FSM ∑ ◫