An H100 advertises 989 teraflops. Generating one token from an 8B model uses roughly 0.3% of that. The gap is not a bug in your code or a missing compiler flag; it is a single ratio, FLOPs per byte moved, and almost every perform…
A 2-billion-parameter model that trades blows with one ten times its size is not an accident of architecture. It is the product of a teacher pouring its full probability distribution into a student, token by token.
Autoregressive models write left to right, one token at a time. Diffusion language models reveal a whole sequence at once and sharpen it over a handful of steps. That single change rewrites the latency math, and in 2025 it stoppe…
Every attention head learns three weight matrices for query, key, and value. A 2026 study trained models up to 1.2B parameters to ask which of them are actually load-bearing, and found that keys and values can share one projectio…
Weight quantisation, KV cache eviction, prompt compression and distillation are treated as four separate engineering disciplines with four separate literatures. They are one problem: choosing a point on a rate-distortion curve. S…
An A100 can execute 312 trillion half-precision operations per second but can only pull about 2 terabytes from memory in that same second. FlashAttention made attention fast not by computing less, but by refusing to touch memory.
A 70B model in FP16 needs 140 GB of memory it spends most of its time waiting to read. Dropping each weight to four bits cuts that to 35 GB, and for years that cut also broke the model. Here is what changed.
Two servers run the same model on the same GPUs. One reports 4,200 tokens per second and is unusable; the other reports 2,600 and feels instant. Throughput is a property of the server, latency is a property of the request, and th…
Pure Mamba was supposed to replace attention. Instead the most efficient open models in 2025 are roughly seven-eighths Mamba and one-eighth attention. The reason is the KV cache, and what each layer can and cannot remember.
The same 1024x1024 screenshot costs 576 visual tokens in one model and 2,880 in another, and the cheap one cannot read the text. The rule that turns pixels into tokens is the least examined hyperparameter in the multimodal stack,…
Every frontier LLM released since 2023 uses some form of Mixture of Experts. The idea is deceptively simple: build a model with hundreds of billions of parameters but activate only a fraction for each token. The result is dense-m…
A GPU loaded with a 13B model can have most of its KV-cache memory sitting idle while requests queue for capacity. PagedAttention and continuous batching reclaim that memory, and the throughput follows.