Arithmetic Intensity: Why Your GPU Is Idle 99% of the Time

An NVIDIA H100 SXM performs 989 trillion BF16 floating-point operations per second. Ask it to generate one token from Llama-3-8B with a batch size of one, and it will perform about 16 billion of them, in roughly 5 milliseconds. That works out to 3.4 teraflops sustained: 0.34 percent of the number on the datasheet.

Nothing is broken. The kernel is not badly written, the model is not badly quantised, and no compiler flag recovers the missing 99.7 percent. The GPU spent essentially the entire 5 milliseconds waiting for 16 gigabytes of weights to arrive from HBM, and it will spend the same 5 milliseconds waiting on the next token too. The arithmetic was never the expensive part.

This one ratio, floating-point operations performed per byte moved, explains more about deep learning performance than any other quantity. It explains why batching is free until suddenly it is not, why FlashAttention is faster despite doing more arithmetic, why grouped-query attention exists, why quantisation speeds up inference more than its FLOP count suggests, and why every accelerator vendor's headline number is the least interesting figure in the datasheet.

Why this matters: Most engineers reach for optimisations by intuition: a faster kernel, a bigger GPU, more parallelism. Half of those interventions cannot help, and knowing which half takes one division. Arithmetic intensity tells you whether you are compute-bound, memory-bound, or overhead-bound, and each regime has a completely different set of fixes. Applying a compute-bound fix to a memory-bound problem is the most common wasted week in ML systems work.

TL;DR

• Arithmetic intensity is FLOPs performed divided by bytes moved to and from off-chip memory. Every kernel has one, and it decides which hardware limit binds.
• The ridge point of an H100 SXM is 989 TFLOPS / 3.35 TB/s ≈ 295 FLOP per byte. Below that intensity you are bandwidth-limited and additional FLOPS are worthless to you.
• Autoregressive decoding at batch size \(B\) has arithmetic intensity of about \(B\) FLOP per byte on the weight matrices. Batch 1 sits at 1, which is 295 times below the ridge, hence 0.34 percent utilisation.
• Peak FLOPS grew about 3.0x every two years while DRAM bandwidth grew 1.6x and interconnect 1.4x (Gholami et al., 2024), so the ridge point has risen steadily and more workloads fall below it every generation.
• Softmax, LayerNorm, GeLU, residual adds and dropout perform almost no arithmetic and move a lot of bytes. Individually trivial, collectively a large share of transformer runtime, which is why fusion is the highest-leverage compiler optimisation.
• FlashAttention is faster while performing more FLOPs, because it never materialises the \(N \times N\) attention matrix in HBM, converting an \(O(N^2)\) memory problem into an \(O(N^2 d^2 / M)\) one (Dao et al., 2022).
• Attention's KV-cache read has arithmetic intensity of roughly 1 FLOP per byte regardless of batch size, because each cached key and value serves exactly one sequence. Batching cannot fix it; GQA, MLA, and cache compression are the only levers.
• The H200 has identical compute to the H100 and 43 percent more bandwidth, which lowers its ridge point to about 206. For memory-bound work, that is a real speedup that appears nowhere in a FLOPS comparison.

At a Glance

flowchart LR
    K["Kernel"] --> AI["Compute arithmetic intensity<br/>FLOPs / bytes moved"]
    AI --> C{"Above ridge point?"}
    C -->|"Yes, AI over 295"| CB["Compute bound<br/>Fix: better tiling, tensor cores, lower precision math"]
    C -->|"No, AI under 295"| MB["Memory bound<br/>Fix: fusion, batching, quantise weights, reuse cache"]
    MB --> OV{"Kernel launch time<br/>dominates?"}
    OV -->|"Yes"| OB["Overhead bound<br/>Fix: CUDA graphs, bigger kernels, less Python"]
    OV -->|"No"| MB2["Genuinely bandwidth limited"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class K blue
    class AI,C,OV purple
    class CB teal
    class MB,MB2 amber
    class OB rose

How the Memory Wall Got Here

The problem is older than deep learning and was named before the first CUDA release. In 1995 Wulf and McKee observed that processor speed and DRAM speed were both improving exponentially at different rates, so the gap between them was itself growing exponentially, and that no amount of cache cleverness would postpone the reckoning indefinitely (Wulf & McKee, 1995, Hitting the Memory Wall: Implications of the Obvious, ACM SIGARCH Computer Architecture News 23(1):20-24). The title's "obvious" was the point: everyone knew the trend lines, and nobody had drawn the conclusion.

The tool that made the trade-off legible arrived in 2009. Williams, Waterman, and Patterson proposed the Roofline model, a single log-log plot with attainable performance on the vertical axis and arithmetic intensity on the horizontal (Williams et al., 2009, Communications of the ACM 52(4):65-76). A diagonal line rising with slope equal to memory bandwidth meets a horizontal line at peak compute, and where they meet is the ridge point. Plot your kernel on that chart and you can see, in one glance, which ceiling you are under and how far from it you are.

timeline
    title From the Memory Wall to Memory-Bound Transformers
    1995 : Wulf and McKee name the memory wall
         : Compute and DRAM improve at different exponential rates
    2009 : Roofline model gives arithmetic intensity a picture
         : Ridge point becomes a design target, not folklore
    2017 : Volta ships tensor cores; FP16 matmul throughput jumps 8x
         : Everything that is not a matmul becomes relatively more expensive
    2020 : Data Movement Is All You Need profiles transformer training
         : Fusing memory-bound ops gives 1.30x on a BERT encoder layer
    2022 : FlashAttention makes attention IO-aware
         : 2-4x faster with 5-20% of the memory, by never writing the N by N matrix
    2023 : PagedAttention and continuous batching attack KV cache waste
         : Serving reframed around memory capacity, not FLOPs
    2024 : AI and Memory Wall quantifies the divergence
         : FLOPS 3.0x per 2 years against DRAM 1.6x and interconnect 1.4x

Volta is the inflection worth dwelling on. Tensor cores raised dense matmul throughput by roughly an order of magnitude in a single generation while leaving memory bandwidth nearly untouched. Every operation that was not a matmul, meaning every normalisation, activation, mask, residual add and softmax, became relatively more expensive overnight. The optimisation problem changed shape, and a lot of tuning intuition built on Pascal stopped transferring.

Gholami et al. put numbers on the long-run divergence: over roughly two decades, peak server FLOPS scaled about 3.0x every two years, while DRAM bandwidth managed 1.6x and interconnect bandwidth 1.4x (Gholami et al., 2024, AI and Memory Wall, arXiv:2403.14123, IEEE Micro). The ridge point is therefore not a constant of nature. It has been climbing for twenty years, and workloads that were compute-bound on a V100 can be memory-bound on an H100 without a single line of code changing.

[IMAGE: Log-log roofline plot with arithmetic intensity (FLOP/byte) on x and attainable TFLOPS on y, showing four rooflines for V100, A100, H100 and B200 with their ridge points marked at 139, 153, 295 and 281. Overlay four workload points: batch-1 decode (AI 1), batch-56 decode (AI 56), training GEMM (AI 500+), and LayerNorm (AI 2.5). Caption: "Three of the four workloads sit on the diagonal, where only bandwidth matters."]

How Arithmetic Intensity Actually Works

The definition and the ridge point

For a kernel that performs \(W\) floating-point operations while moving \(Q\) bytes between off-chip memory and the chip, arithmetic intensity is

Attainable performance is then bounded by both ceilings at once:

with \(P_{\text{peak}}\) the peak compute rate and \(\beta\) the memory bandwidth. The crossover, the ridge point, is where the two terms are equal:

For an H100 SXM at 989 TFLOPS dense BF16 and 3.35 TB/s of HBM3, that is 295 FLOP per byte. The interpretation is blunt: unless a kernel performs at least 295 floating-point operations for every byte it reads or writes, the tensor cores will be idle waiting for data, and the fraction of peak you can reach is exactly \(I / 295\).

Note what \(I_{\text{ridge}}\) is a ratio of. Buying a chip with more FLOPS and the same bandwidth raises the ridge point, making more of your workload memory-bound. This is why the H200, with identical Hopper compute and HBM3e at 4.8 TB/s, is genuinely faster for inference than the H100 despite being the same compute silicon: its ridge point falls to about 206, and every memory-bound kernel gets 43 percent more bandwidth.

Where each transformer operation lands

The arithmetic intensity of the operations in a transformer spans four orders of magnitude.

Matrix multiply, both operands large. For \(C = AB\) with \(A\) of shape \(M \times K\) and \(B\) of shape \(K \times N\), the work is \(2MKN\) FLOPs and the traffic, in the best case where everything is read once, is \(2(MK + KN + MN)\) bytes at BF16. For square \(n \times n\) matrices this gives \(I \approx n/3\), so intensity grows linearly with the matrix dimension. A 4096-cube matmul lands around 1,365 FLOP per byte, comfortably compute-bound. This is the only operation in the entire stack that is reliably above the ridge point, and it is why transformer training at reasonable batch sizes is a compute-bound workload.

[IMAGE: Horizontal log-scale strip chart of arithmetic intensity for eight transformer operations, from residual add at 0.17 to a 4096-cube GEMM at 1365, with a vertical red line at the H100 ridge point of 295. Caption: "Exactly one operation in a transformer reliably sits on the compute-bound side of the line."]

Matrix-vector multiply, which is what decoding is. Take \(y = Wx\) with \(W\) of shape \(N \times K\) in BF16 and \(x\) holding \(B\) columns, one per sequence in the batch. Work is \(2NKB\) FLOPs. Traffic is dominated by reading \(W\) once: \(2NK\) bytes. So

The arithmetic intensity of a decode step equals the batch size. Not approximately; exactly, to the precision of ignoring the activations. Batch 1 gives an intensity of 1 against a ridge point of 295. There is no kernel, no compiler, and no amount of tuning that recovers more than 1/295 of peak, because the information-theoretic minimum traffic is already being moved.

Elementwise and reduction operations. A LayerNorm reads a tensor, computes a mean and a variance, and writes a tensor. Per element it moves 4 bytes and performs on the order of 10 FLOPs, so \(I \approx 2.5\). GeLU is worse. A residual add is \(I = 0.33\). These operations are individually negligible in FLOP terms and collectively significant in time, which is the entire justification for kernel fusion: fusing LayerNorm, the following GeLU, and the residual add into one kernel does not reduce the arithmetic at all, and cuts the memory traffic by a factor of three.

Ivanov et al. profiled exactly this on transformer training and found data movement, not arithmetic, to be the binding constraint. Fusing memory-bound operations and choosing better data layouts gave 1.30x on a BERT encoder layer and 1.19x on full BERT training against the best available frameworks (Ivanov et al., 2021, Data Movement Is All You Need, arXiv:2007.00072, MLSys). The paper's framing is worth adopting: they treat the transformer as a data-movement problem with some arithmetic attached, rather than the reverse.

Attention, and why FlashAttention wins by doing more work

Standard attention computes \(S = QK^\top\), then \(P = \text{softmax}(S)\), then \(O = PV\). The textbook implementation writes \(S\) and \(P\) to HBM. For sequence length \(N\) and head dimension \(d\), that is \(O(N^2)\) bytes of traffic against \(O(N^2 d)\) FLOPs, giving an intensity of about \(d\), which for \(d = 128\) is far below the ridge point. Worse, the memory capacity cost is quadratic, which is what makes long context expensive before it makes it slow.

FlashAttention restructures the computation into tiles that fit in SRAM, using the online-softmax trick to accumulate the normalisation incrementally so the full \(S\) matrix is never materialised anywhere except on-chip (Dao et al., 2022, FlashAttention, arXiv:2205.14135, NeurIPS 2022). HBM traffic drops from \(O(N^2)\) to \(O(N^2 d^2 / M)\), where \(M\) is SRAM size.

The rescaling arithmetic means FlashAttention performs more floating-point operations than the naive version. It is 2 to 4 times faster and uses 5 to 20 percent of the memory anyway. That inversion, where the algorithm doing more arithmetic is decisively faster, is the clearest possible demonstration that FLOPs are the wrong unit of account.

[IMAGE: Side-by-side memory-traffic diagrams for standard attention and FlashAttention at sequence length 4096. Left: Q, K, V read from HBM, the N-by-N score matrix written to HBM and read back twice, output written. Right: Q, K, V read in tiles, all intermediates confined to an on-chip SRAM box, output written once. Annotate total HBM bytes under each. Caption: "The right-hand version performs more floating-point operations and moves an order of magnitude fewer bytes."]

The KV cache during decoding has a nastier property. Each cached element participates in exactly one multiply-accumulate against the current query, which is 2 FLOPs, and occupies 2 bytes at BF16. Its arithmetic intensity is therefore about 1 FLOP per byte, independent of the batch size. Each cached key and value belongs to exactly one sequence, so batching creates no reuse. This is the structural reason why grouped-query attention and multi-head latent attention exist: unable to raise the intensity, they cut \(Q\) instead.

Seeing It in Motion

sequenceDiagram
    participant H as HBM at 3.35 TB per second
    participant S as SRAM and registers
    participant T as Tensor cores at 989 TFLOPS

    Note over H,T: One decode step, batch size 1, Llama-3-8B
    H->>S: Read layer 1 weights (0.5 GB)
    S->>T: Multiply-accumulate (0.5 GFLOP)
    T-->>S: Result vector, 8 KB
    Note over T: Cores idle 99.7% of this interval
    H->>S: Read layer 1 KV cache
    S->>T: Attention over cached keys and values
    Note over H,T: Repeat for all 32 layers
    Note over H,T: Total 16 GB moved, 16 GFLOP performed, intensity 1

The sequence above is the whole problem in one picture. Every arrow from HBM is expensive and every arrow to the tensor cores is nearly free, and the ratio between them does not improve no matter how good the kernel is, because the weights must be read and there is only one token's worth of work to do with them.

Batching changes exactly one thing: it amortises the weight read across more tokens. The weight traffic is identical for batch 1 and batch 56; the arithmetic is 56 times larger. That is why throughput per GPU rises almost linearly with batch size in the memory-bound regime and then flattens abruptly at the ridge point, and why the correct mental model of a serving system is a machine trying to keep the batch large enough to escape the diagonal.

graph TD
    subgraph Weights["Weight traffic, amortised by batching"]
        W1["Read W once per step"] --> W2["Use for B tokens"]
        W2 --> W3["Intensity equals B"]
    end
    subgraph KV["KV cache traffic, not amortised"]
        K1["Read cache once per step"] --> K2["Each entry serves 1 sequence"]
        K2 --> K3["Intensity stays near 1"]
    end
    W3 --> D{"Which dominates?"}
    K3 --> D
    D -->|"Short context"| SD["Weights dominate; batch harder"]
    D -->|"Long context"| LD["KV dominates; cut cache size with GQA or MLA"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff

    class W1,W2,W3 blue
    class K1,K2,K3 rose
    class D purple
    class SD teal
    class LD amber

[IMAGE: Two stacked bar charts of per-decode-step HBM traffic for Llama-3-8B at batch 56, one at 2k context and one at 8k context, splitting weights (15 GiB, constant) from KV cache (14 GiB and 56 GiB). Caption: "The same model, the same batch, four times the context: the workload changes from weight-dominated to cache-dominated, and the correct optimisation changes with it."]

By the Numbers

Sources: NVIDIA published datasheet figures for each part, dense (non-sparsity) tensor-core throughput. The widely quoted 1,979 TFLOPS for H100 and 4,500 for B200 are the 2:4 structured-sparsity numbers and do not apply to dense models. Ridge points are computed as peak divided by bandwidth.

Intensities in the second table are derived from the operation definitions rather than measured; treat them as the ceiling a perfect kernel would approach.

A Concrete Example

Llama-3-8B on a single H100 SXM. Configuration: 8.03 billion parameters, 32 layers, hidden size 4096, 32 query heads and 8 key-value heads with head dimension 128, weights in BF16.

Step 1: weight traffic per decode step. Every parameter must be read from HBM once per forward pass.

At 3.35 TB/s that is \(16.1 / 3350 = 4.79\) ms, before any arithmetic happens.

Step 2: the arithmetic, for comparison. A forward pass costs about \(2N\) FLOPs per token:

At 989 TFLOPS that is 0.0163 ms. The ratio of memory time to compute time is \(4.79 / 0.0163 = 294\), which is the ridge point, arrived at from the other direction. Sustained utilisation at batch 1 is \(1/295 = 0.34\) percent, and the theoretical ceiling is \(1000/4.79 = 209\) tokens per second. Measured single-stream throughput for this model on an H100 typically lands somewhere in the 130 to 180 range, the shortfall being kernel launch overhead and imperfect overlap, so the roofline is a real ceiling rather than a loose one.

Step 3: add the KV cache. With grouped-query attention at 8 KV heads and head dimension 128, per token per layer the cache holds

Across 32 layers that is 128 KiB per token of context. At batch 1 with 8,192 tokens of context, the cache is 1.07 GB and must be read every step, pushing traffic to 17.2 GB and the ceiling down to 195 tokens per second.

Step 4: batch to escape the diagonal. At batch 56, weight traffic is unchanged at 16.1 GB, but the arithmetic is 56 times larger and intensity rises to 56, so utilisation reaches \(56/295 = 19\) percent. Ignoring the cache, 56 tokens now emerge every 4.79 ms, which is 11,690 tokens per second: a 56x throughput gain for zero additional weight traffic. This is the single most important fact in LLM serving economics.

Step 5: run out of memory instead. The batch cannot grow indefinitely, and the binding constraint is capacity, not bandwidth. An 80 GB card holds 74.5 GiB; the weights take 15.0 GiB, leaving about 59 GiB for cache. At 128 KiB per token:

At 8,192 tokens of context per sequence, that is about 59 sequences, and 56 once you leave headroom for activations and allocator slack. Reaching the ridge point would require batch 295, which needs roughly 36 GiB of cache per 1,000 tokens of context and is impossible on this GPU at any useful context length.

Step 6: notice what now dominates. At batch 56 with 8k context, per-step traffic is 16.1 GB of weights plus 56 GiB, or 60.1 GB, of KV cache. The cache is now nearly four times the weight traffic, and it does not amortise across the batch. Total per-step time is \(76.2 / 3350 = 22.7\) ms for 56 tokens, or about 2,470 tokens per second.

[IMAGE: Line chart of Llama-3-8B throughput on one H100 against batch size from 1 to 128, with two curves: the roofline ceiling ignoring KV cache (rising linearly then flattening at 295) and the achievable curve including 8k-context KV traffic (rising then bending over well before the ridge). Shade the region past batch 59 as "does not fit in 80 GB". Caption: "The batch never reaches the ridge point, because capacity runs out first."]

Step 6 is the payoff. A practitioner looking at 2,470 tokens per second and reaching for a faster matmul kernel is optimising 21 percent of the traffic. The other 79 percent is the KV cache, and the interventions that touch it are grouped-query attention, multi-head latent attention, cache quantisation, eviction, and prefix sharing. The arithmetic above takes ten minutes and redirects a month of work.

Where It Breaks

The third regime nobody plots

Roofline has two ceilings; real systems have three. When kernels are small, neither compute nor bandwidth binds, because the GPU is idle between kernels waiting for the CPU to enqueue the next one. A launch costs a few microseconds; a batch-1 LayerNorm on a 4096-dimensional vector takes less than that. Horace He's framing of compute, bandwidth, and overhead as three distinct regimes is the practical extension of roofline, and the diagnostic is simple: if profiled GPU time is far below wall-clock time, you are overhead-bound and no kernel optimisation will help (He, 2022, Making Deep Learning Go Brrrr From First Principles). CUDA graphs, larger fused kernels, and getting Python out of the inner loop are the fixes, and they are unrelated to everything else in this article.

Peak is not attainable, and the shortfall is structured

The roofline's horizontal ceiling assumes tensor cores are perfectly fed. Two quantisation effects, in the tiling sense rather than the numeric sense, stop that from happening.

Tile quantisation. GEMM kernels decompose the output into fixed tiles, commonly 128 by 128 or 256 by 128. An output matrix whose dimensions are not multiples of the tile size pads the final tile with waste, and the hardware computes the padding at full cost. A GEMM with \(N = 257\) against a 128-wide tile does the work of \(N = 384\).

Wave quantisation. Thread blocks execute in waves across the streaming multiprocessors. With 132 SMs on an H100, a kernel launching 133 thread blocks runs two waves and takes twice as long as one launching 132, for 0.8 percent more work. NVIDIA's performance guides document both effects with measured curves, and they explain why throughput as a function of batch size is a staircase rather than a line (NVIDIA, Matrix Multiplication Background User's Guide).

[IMAGE: Staircase plot of measured GEMM throughput against the N dimension from 240 to 400 in steps of 1, on a 128-wide tile, showing flat treads and sharp risers at multiples of 128. Overlay a second staircase for wave quantisation across 132 SMs. Caption: "Throughput as a function of shape is a staircase; picking N = 257 buys the cost of N = 384."]

Capacity and bandwidth are different limits, and people conflate them

"Memory-bound" is used for two unrelated problems. A kernel can be bandwidth-bound, meaning it waits on the rate of transfer, or a system can be capacity-bound, meaning the working set does not fit and something must be evicted or sharded. The batch-56 ceiling in the worked example is a capacity limit; the 4.79 ms weight read is a bandwidth limit. They have different fixes, and quantisation is nearly the only intervention that helps both at once, which is a large part of why it is so popular.

The cache hierarchy is missing from the model

Classic roofline treats memory as a single level. Real GPUs have registers, SRAM, L2, and HBM, with an order of magnitude of bandwidth between adjacent levels. Every serious optimisation, FlashAttention included, is really about which level the traffic occurs at, and a single-level roofline cannot express that. Hierarchical roofline models exist and are used in HPC; they have not become standard practice in ML.

Sparsity and low precision move the goalposts

The 2:4 structured sparsity numbers on NVIDIA datasheets double the compute ceiling and leave bandwidth untouched, which raises the ridge point and makes matters worse for memory-bound work. FP8 and FP4 are different: they double or quadruple the compute ceiling and halve or quarter the bytes per parameter, so both terms move. That is why low-precision inference delivers speedups that exceed what its FLOP count predicts, and why the quantisation literature is really a bandwidth literature wearing a numerics disguise.

Mixture-of-experts breaks the batching argument

The batch-amortisation result assumes every token uses every weight. In a sparse MoE layer, tokens in a batch route to different experts, so a batch of 64 might touch all 256 experts and read all of their weights while performing only 64 tokens' worth of arithmetic on each. Effective intensity falls back toward the batch-1 case, and this, not the FLOP count, is the reason MoE serving is memory-hostile in ways its active-parameter count hides.

Alternative Designs

Speculative decoding is the entry most worth understanding through this lens. It is usually explained as "a small model drafts and a big model checks", which makes it sound like a compute saving. It is the opposite. Verifying \(k\) draft tokens in one forward pass reads the weights once and performs \(k\) tokens' worth of arithmetic, so it raises arithmetic intensity from 1 to \(k\) in exactly the way batching does, except it works when you have only one request. The speedup comes from spending the idle FLOPs you were already paying for.

How It Is Used in Practice

Every production inference stack is, structurally, a machine for raising arithmetic intensity, whatever else its documentation says it is for.

Continuous batching exists because static batching leaves the batch dimension small whenever sequences finish at different times, and a small batch is a low intensity. vLLM's contribution was as much a scheduling insight as a memory one: keep the batch full at every iteration and the diagonal is escaped for more of the time.

PagedAttention attacks the capacity limit that caps the batch. Fragmentation in a naive contiguous cache allocator wastes a large fraction of HBM, and every wasted byte is a sequence that cannot join the batch. Raising achievable batch size raises intensity, which is why a memory-management technique shows up as a throughput number.

Prefill and decode disaggregation takes the analysis to its conclusion. Prefill processes thousands of tokens at once and is compute-bound, comfortably above the ridge point. Decode processes one token per sequence and is memory-bound, far below it. These are different workloads with opposite hardware preferences, and running them on the same GPUs forces a compromise that suits neither. Splitting them across separate pools lets each be provisioned against the ceiling that actually binds it.

[IMAGE: Two-panel roofline showing prefill and decode as separate points for the same model and hardware. Prefill sits at intensity ~2000 near the compute ceiling; decode sits at intensity 56 on the diagonal. Draw an arrow between them labelled "same GPU pool, opposite constraints". Caption: "Prefill and decode are different workloads wearing the same model's weights, which is the entire argument for disaggregated serving."]

Cost modelling should start here. A serving cost estimate built from FLOPs and a GPU's peak throughput will be wrong by two orders of magnitude for decode. Estimating from bytes moved and memory bandwidth lands within tens of percent, which is close enough to plan capacity from. When a vendor quotes tokens per second per dollar, the first question is the batch size and context length, because those two numbers determine which ceiling the quote is measured against.

Insights Worth Remembering

1. Arithmetic intensity of a decode step equals the batch size. Not approximately, exactly. This single identity determines the shape of every LLM serving cost curve, and it explains why batch-1 latency and high-throughput serving are genuinely different engineering problems rather than two points on one dial.

2. More FLOPS can make your workload slower, relatively. The ridge point is peak compute divided by bandwidth, so a chip that improves compute more than bandwidth moves more of your work into the memory-bound region. The H200 is faster than the H100 for inference precisely because it did the opposite.

3. FlashAttention doing more arithmetic while running faster is not a paradox; it is the thesis. Once you accept that bytes are the currency, trading FLOPs for bytes is obviously a good deal, and a lot of algorithm design that looks strange becomes routine.

4. Fusion is free performance and nothing else in the stack is. Fusing memory-bound operations reduces traffic without changing the mathematics, the numerics, or the model. Every other intervention on this list costs accuracy, latency, memory, or engineering time.

5. The KV cache is the term batching cannot fix. Weight traffic amortises across a batch; cache traffic does not, because each entry serves exactly one sequence. Any long-context serving problem eventually becomes a cache problem, and no amount of batching postpones it.

6. Do the division before the optimisation. FLOPs divided by bytes takes a minute on paper and tells you which of three disjoint sets of fixes can possibly help. The most common failure in ML performance work is not choosing a bad optimisation; it is choosing a good optimisation for the wrong regime.

7. Quantisation is a bandwidth technique. Its accuracy story is what papers discuss and its byte count is what makes it fast. INT4 weights move a quarter of the bytes of BF16, which is a 4x reduction in the dominant term of batch-1 decode, and that is where nearly all the speedup comes from.

8. Peak FLOPS is the least informative number on a datasheet. Bandwidth, capacity, and the ratio between compute and bandwidth predict deep learning performance far better, and none of the three is the number in the headline.

Open Questions

Will the ridge point keep climbing? Measured over two decades the divergence is clear: 3.0x compute against 1.6x bandwidth per two years. HBM3e and HBM4 narrow the gap somewhat, and packaging advances may narrow it further. Whether the trend bends is an open empirical question; nothing in the physics guarantees either outcome, and the answer determines whether architectures must keep being redesigned around bandwidth.

Can architectures be co-designed for intensity from the start? Multi-head latent attention was designed partly to shrink the KV cache, and it works. Whether there is a general design methodology, as opposed to a series of point fixes, is unresolved. The question is whether "arithmetic intensity" belongs in the architecture search space alongside parameter count and depth.

Does memory-centric hardware change the answer? Processing-in-memory and wafer-scale designs attack the ratio directly by moving computation to the data. Cerebras and several PIM efforts have shipped working systems. Whether they can match the software ecosystem and utilisation of GPUs on real workloads remains, as of early 2026, unproven at scale rather than disproven.

How much of the gap between roofline and measured performance is recoverable? Measured single-stream decode typically reaches 60 to 85 percent of the roofline ceiling. The remainder is launch overhead, imperfect overlap, and tile effects. Nobody has published a careful decomposition of that shortfall across models and hardware, and it would be a genuinely useful piece of work.

Is there a good hierarchical roofline for deep learning? The single-level model cannot express the SRAM-versus-HBM distinction that FlashAttention exploits, which means the field's standard analysis tool cannot describe its most important kernel optimisation. Hierarchical variants exist in HPC and have not been adapted into ML tooling.

Sources and Further Reading

1. Wulf, W. A., & McKee, S. A. (1995). "Hitting the Memory Wall: Implications of the Obvious." ACM SIGARCH Computer Architecture News, 23(1), 20-24. DOI:10.1145/216585.216588
2. Williams, S., Waterman, A., & Patterson, D. (2009). "Roofline: An Insightful Visual Performance Model for Multicore Architectures." Communications of the ACM, 52(4), 65-76. DOI:10.1145/1498765.1498785
3. Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., & Keutzer, K. (2024). "AI and Memory Wall." IEEE Micro, 44(3), 33-39. arXiv:2403.14123
4. Ivanov, A., Dryden, N., Ben-Nun, T., Li, S., & Hoefler, T. (2021). "Data Movement Is All You Need: A Case Study on Optimizing Transformers." MLSys. arXiv:2007.00072
5. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." NeurIPS. arXiv:2205.14135
6. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP. arXiv:2309.06180
7. He, H. (2022). "Making Deep Learning Go Brrrr From First Principles." horace.io/brrr_intro.html
8. NVIDIA. "Matrix Multiplication Background User's Guide" and "GPU Performance Background User's Guide." NVIDIA Deep Learning Performance Documentation. docs.nvidia.com
9. NVIDIA. "NVIDIA H100 Tensor Core GPU Datasheet" and "NVIDIA A100 80GB Tensor Core GPU Datasheet." nvidia.com/en-us/data-center/h100
10. Leviathan, Y., Kalman, M., & Matias, Y. (2023). "Fast Inference from Transformers via Speculative Decoding." ICML. arXiv:2211.17192
11. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., & Zhang, H. (2024). "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving." OSDI. arXiv:2401.09670
12. Dao, T. (2024). "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." ICLR. arXiv:2307.08691

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.