Tagged “kv-cache”

7 posts.

All Model Architecture19 Training & Alignment22 Inference & Serving18 Agents & Orchestration11 Reasoning & Evaluation26 Safety, Security & Governance7 Platforms & Practice20

Inference & Serving 21 min

Cache-Augmented Generation: When Preloaded KV-Caches Replace Retrieval Pipelines

Retrieval-augmented generation fetches documents at query time, scores them, and hopes the retriever got it right. Cache-Augmented Generation sidesteps the entire pipeline by preloading knowledge into the model's KV-cache before …

kv-cache cache-augmented-generation transformer-inference rag-alternative ∑ ◫

Model Architecture 20 min

Do Transformers Need Three Projections? Rethinking Q, K, and V

Every attention head learns three weight matrices for query, key, and value. A 2026 study trained models up to 1.2B parameters to ask which of them are actually load-bearing, and found that keys and values can share one projectio…

transformers attention kv-cache inference ∑ ◫

Inference & Serving 27 min

Everything Is Lossy Compression: A Rate-Distortion View of Quantisation, KV Caches, and Distillation

Weight quantisation, KV cache eviction, prompt compression and distillation are treated as four separate engineering disciplines with four separate literatures. They are one problem: choosing a point on a rate-distortion curve. S…

information-theory quantisation kv-cache distillation ∑ ◫

Inference & Serving 24 min

Goodput, Not Throughput: The Metric That Decides Whether Your LLM Deployment Works

Two servers run the same model on the same GPUs. One reports 4,200 tokens per second and is unusable; the other reports 2,600 and feels instant. Throughput is a property of the server, latency is a property of the request, and th…

inference serving latency goodput ∑ ◫

Model Architecture 22 min

Multi-Head Latent Attention: How DeepSeek Compressed the KV Cache Without Losing Quality

Every token an LLM generates forces it to reload the keys and values of every token that came before. Multi-Head Latent Attention rewrites that trade by caching one compressed vector instead of dozens of separate heads, cutting m…

llm-architecture attention-mechanisms deepseek kv-cache ∑ ◫

Inference & Serving 21 min

PagedAttention and Continuous Batching: How vLLM Stopped Wasting Your GPU

A GPU loaded with a 13B model can have most of its KV-cache memory sitting idle while requests queue for capacity. PagedAttention and continuous batching reclaim that memory, and the throughput follows.

inference vllm kv-cache gpu-serving ∑ ◫

Inference & Serving 24 min

Running Language Models on a Phone: Memory Bandwidth, NPUs, and the Few-Billion-Parameter Ceiling

Phones ship NPUs rated in trillions of operations per second, yet the speed at which a reply appears is set by how fast LPDDR memory can hand a couple of gigabytes of weights to the processor, over and over. This post derives the…

on-device-and-edge-ai on-device-ai llm-inference memory-bandwidth ∑ ◫