The Memory Wall and Arithmetic Intensity

A 100-billion-parameter model running on an A100 GPU is doing, in aggregate, an astronomical number of floating-point multiplications per second. Yet token generation is often limited not by those multiplications but by how fast the GPU can shuttle numbers in from DRAM. The chip is fast; the wires feeding it are the bottleneck. This gap between compute throughput and memory bandwidth is called the memory wall, and understanding it precisely changes how you think about every optimisation decision from quantisation to batching to operator fusion.

What the Roofline Model Actually Says

The roofline model is a two-line performance ceiling drawn on a log-log plot of achieved performance (FLOP/s) against arithmetic intensity (FLOP/byte). Arithmetic intensity for a kernel is:

I = total floating-point operations
    ─────────────────────────────────
    total bytes read from + written to memory

Every chip has two hard limits:

• Peak compute throughput P (FLOP/s), set by transistor count and clock speed.
• Peak memory bandwidth B (bytes/s), set by the memory bus width and frequency.

The ridge point of the roofline is the intensity threshold I = P / B. Below I, the kernel is memory-bound: performance scales with bandwidth, not compute. Above I*, the kernel is compute-bound: performance scales with FLOP/s, not bandwidth.

For an NVIDIA V100, P ≈ 125 TFLOP/s (FP16 with Tensor Cores) and B ≈ 900 GB/s, giving I ≈ 139 FLOP/byte. On the A100, P ≈ 312 TFLOP/s and B ≈ 2,000 GB/s (HBM2e), giving I ≈ 156 FLOP/byte.

The practical implication: a kernel must perform roughly 140-160 multiply-accumulates per byte it touches just to keep the compute units busy. Most neural network operations do nowhere near that.

The Intensity of Real Operations

Here is where the gap becomes concrete. NVIDIA's deep-learning performance guide (see Further Reading) gives measured arithmetic intensities for common operations on a V100:

The batch=1 GEMM row is the one that matters for autoregressive inference. When generating tokens one at a time, the weight matrix (e.g., a 4096 x 4096 projection) is loaded from HBM for each forward pass, but only a single vector (the current hidden state) is multiplied against it. The ratio of arithmetic to bytes is tiny. The GPU is mostly waiting for data.

A useful back-of-envelope: for a decoder-only transformer with hidden size H, an attention projection of shape (H, H) holds H^2 parameters. In FP16 that is 2H^2 bytes. The GEMM for a single token does 2H^2 floating-point operations (one multiply and one accumulate per parameter). Intensity = 2H^2 / 2H^2 = 1 FLOP/byte, which is roughly 140x below the ridge point. The chip is therefore running at about 1/140^th of its peak FLOP/s during single-token generation.

Why Batch Size Is a Lever, Not Just a Throughput Trick

When you increase batch size from 1 to B, the same weight matrix is read once from HBM and used for B multiplications. Intensity scales as:

I(batch=B) ≈ B · (2 · seq_len · H) / (2 · H^2 + 2 · B · seq_len · H)
           ≈ B / H   (when B·seq_len << H, i.e., small sequences)

For H = 4096 and the ridge point at 140, you need B ≈ 140 · H / H = 140 to approach compute-boundedness on a weight-only read. This is why throughput-focused serving systems (like vLLM in continuous batching mode) fight hard to pack as many requests as possible into each forward pass: they are converting a memory-bandwidth problem into a compute problem.

Prefill (processing the prompt) is a different story. Here the input is a full token matrix of shape (seq_len, H), and the GEMM is (seq_len, H) x (H, H). For seq_len = 1024 and H = 4096, intensity is roughly 1024 / 4096 ≈ 0.25 of ridge-point crossover. Still somewhat memory-bound, but much closer to compute-bound than single-token decode.

The KV Cache Compounds the Problem

During autoregressive generation, each new token must attend to all previous tokens. The key and value tensors from earlier steps are cached in HBM rather than recomputed (KV cache). For a model with L layers, H hidden size, and context length T, the KV cache holds 2 · L · T · H values (two matrices, keys and values, across all layers). At FP16 this is 4 · L · T · H bytes.

For LLaMA-2-70B (L=80, H=8192) at T=4096, the KV cache alone occupies roughly 4 · 80 · 4096 · 8192 bytes ≈ 10.7 GB. Reading this entire cache each generation step, just to produce a single new token, costs ~10.7 GB of memory bandwidth. At 2 TB/s (A100 HBM), that is about 5 ms per step before any compute. This is why long contexts hurt latency so directly: the bandwidth cost grows linearly with context length while compute per step barely changes.

FlashAttention (Dao et al., 2022) attacks exactly this by reordering the attention computation to tile over HBM blocks, keeping intermediate softmax statistics in on-chip SRAM and never writing full attention matrices back to HBM. The IO complexity drops from O(N^2) to O(N^2 / M) where M is SRAM capacity, turning a memory-bandwidth-dominated kernel into something far more efficient.

When It Falls Down

Multi-query and grouped-query attention shift the bottleneck. MQA and GQA reduce the KV cache size by sharing key/value heads, which reduces bandwidth pressure during decode. But if the model becomes compute-bound for other reasons (e.g., very large batch sizes), this saving matters less than the reduction in model quality suggests.

The roofline ignores latency hiding. A GPU can overlap memory transfers with computation through warp scheduling. A kernel operating at, say, 50% of theoretical bandwidth-bound performance may not be fixable by simple fusion; the issue may be instruction-level dependencies rather than bandwidth saturation. Profiling with Nsight Systems distinguishes these cases.

Operator fusion breaks the simple model. When two memory-bound kernels are fused, the combined kernel reads data once and avoids the intermediate HBM write-read round trip. The arithmetic intensity of the fused kernel can be much higher than either in isolation. This is why PyTorch 2.0's torch.compile can produce 2x+ speedups on element-wise chains with no additional arithmetic: it raises effective intensity by eliminating redundant memory traffic, not by doing more computation.

Quantisation is not free bandwidth. Halving parameter size from FP16 to INT8 roughly halves bandwidth consumed loading weights, so it roughly doubles the intensity of weight-load-dominated kernels. But dequantisation adds compute back in, and the threshold at which the kernel crosses from memory-bound to compute-bound changes. At very low precision (INT4), dequantisation overhead can dominate if not handled carefully.

SRAM capacity is finite. FlashAttention's tiling approach works as long as the tile size fits in SRAM. On the A100, SRAM (shared memory + registers per SM) is on the order of a few MB total. For very long sequences or very large head dimensions, tile sizes must shrink, reducing the effective benefit.

Further Reading

• NVIDIA. "GPU Performance Background." Deep Learning Performance Guide. https://docs.nvidia.com/deeplearning/performance/dl-performance-gpu-background/index.html
• NVIDIA. "Matrix Multiplication Background." Deep Learning Performance Guide. https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html
• Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Re, C. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." arXiv:2205.14135 (2022). https://arxiv.org/abs/2205.14135
• He, H. "Making Deep Learning Go Brrrr From First Principles." https://horace.io/brrr_intro.html