Tagged “kv-cache”

5 posts.

All Model Architecture13 Training & Alignment8 Inference & Serving11 Agents & Orchestration10 Reasoning & Evaluation7 Safety, Security & Governance3 Platforms & Practice11

Inference & Serving 21 min

Cache-Augmented Generation: When Preloaded KV-Caches Replace Retrieval Pipelines

Retrieval-augmented generation fetches documents at query time, scores them, and hopes the retriever got it right. Cache-Augmented Generation sidesteps the entire pipeline by preloading knowledge into the model's KV-cache before …

kv-cache cache-augmented-generation transformer-inference rag-alternative ∑ ◫

Model Architecture 20 min

Do Transformers Need Three Projections? Rethinking Q, K, and V

Every attention head learns three weight matrices for query, key, and value. A 2026 study trained models up to 1.2B parameters to ask which of them are actually load-bearing, and found that keys and values can share one projectio…

transformers attention kv-cache inference ∑ ◫

Inference & Serving 24 min

Goodput, Not Throughput: The Metric That Decides Whether Your LLM Deployment Works

Two servers run the same model on the same GPUs. One reports 4,200 tokens per second and is unusable; the other reports 2,600 and feels instant. Throughput is a property of the server, latency is a property of the request, and th…

inference serving latency goodput ∑ ◫

Model Architecture 22 min

Multi-Head Latent Attention: How DeepSeek Compressed the KV Cache Without Losing Quality

Every token an LLM generates forces it to reload the keys and values of every token that came before. Multi-Head Latent Attention rewrites that trade by caching one compressed vector instead of dozens of separate heads, cutting m…

llm-architecture attention-mechanisms deepseek kv-cache ∑ ◫

Inference & Serving 21 min

PagedAttention and Continuous Batching: How vLLM Stopped Wasting Your GPU

A GPU loaded with a 13B model can have most of its KV-cache memory sitting idle while requests queue for capacity. PagedAttention and continuous batching reclaim that memory, and the throughput follows.

inference vllm kv-cache gpu-serving ∑ ◫