Tagged “attention”

Inference & Serving 29 min

Arithmetic Intensity: Why Your GPU Is Idle 99% of the Time

An H100 advertises 989 teraflops. Generating one token from an 8B model uses roughly 0.3% of that. The gap is not a bug in your code or a missing compiler flag; it is a single ratio, FLOPs per byte moved, and almost every perform…

neural-plumbing gpu performance kernels ∑ ◫

Model Architecture 23 min

Context Rot: Why Bigger Context Windows Don't Mean Better Retrieval

A million-token window promises perfect recall of everything you feed it. Controlled tests on 18 frontier models show recall degrading steadily, unevenly, and well before the window fills, a pattern researchers now call context rot.

llm long-context rag context-engineering ∑ ◫

Model Architecture 20 min

Do Transformers Need Three Projections? Rethinking Q, K, and V

Every attention head learns three weight matrices for query, key, and value. A 2026 study trained models up to 1.2B parameters to ask which of them are actually load-bearing, and found that keys and values can share one projectio…

transformers attention kv-cache inference ∑ ◫

Inference & Serving 21 min

FlashAttention and the Memory Wall: Why Attention Was Never Compute-Bound

An A100 can execute 312 trillion half-precision operations per second but can only pull about 2 terabytes from memory in that same second. FlashAttention made attention fast not by computing less, but by refusing to touch memory.

flashattention gpu attention transformers ∑ ◫

Model Architecture 4 min

The Length of a Thought: Why Context Windows Became the New Battleground

A 512-token model could read a paragraph. A million-token model can read a novel. The journey between those two numbers is a story about quadratic cost, clever approximations, and what it means for a machine to "remember."

transformers attention context scaling

Model Architecture 23 min

The Residual Stream: The Transformer's Shared Memory Bus

Stop reading the transformer as a pipeline of 96 layers, each transforming the output of the last. Read it as one shared communication channel that every attention head and MLP merely edits. This one reframing, formalised by inte…

transformer-anatomy interpretability residual-stream attention ∑ ◫

Model Architecture 32 min

The Wavelength Problem: How One Number Decides a Model's Context Length

Llama 2 and Llama 3 share an attention implementation almost line for line. One constant differs: the RoPE base went from 10,000 to 500,000. That number sets the entire wavelength spectrum of the model's position signal, and it h…

positional-encoding rope long-context transformers ∑ ◫

Model Architecture 18 min

Trainable Sparse Attention: When the Model Learns What to Skip

For years, sparse attention was a bolt-on applied after training to shrink the KV cache. NSA, MoBA, and DeepSeek's DSA flip that order, training the sparsity end to end and reporting up to 11x faster decoding without losing accuracy.

attention long-context sparse-attention transformers ∑ ◫