Compute-Bound vs Memory-Bound Kernels

A 100-TFLOP/s GPU can spend 90% of its time waiting for data. Add more matrix multiply units and nothing changes. The hardware is already idle. This is not a GPU problem; it is a kernel classification problem, and getting it wrong sends engineers chasing the wrong bottleneck.

Every kernel falls on a spectrum between two extremes: compute-bound (the ALUs are saturated, memory can keep up) and memory-bound (the ALUs are idle, waiting for bytes to arrive). The distinction is not academic. It dictates which optimisation strategies are even physically capable of helping.

Arithmetic Intensity and the Roofline Model

The key quantity is arithmetic intensity (AI), the ratio of floating-point operations to bytes of memory traffic:

AI = FLOPs executed / bytes read+written from DRAM
     (units: FLOP/byte)

GPUs have two hard ceilings. Let peak compute be P (FLOP/s) and peak memory bandwidth be B (byte/s). The maximum throughput a kernel can achieve is:

Attainable performance = min(P, AI × B)

This is the Roofline model. Plotting attainable performance against AI gives a shape with two regions:

• Left of the ridge point (AI < P/B): the kernel is memory-bound. Doubling compute units does nothing; you need more bandwidth or less data movement.
• Right of the ridge point (AI > P/B): the kernel is compute-bound. Reducing memory pressure yields little; you need faster ALUs or more parallelism.

On an NVIDIA A100 SXM, peak FP16 tensor-core throughput is roughly 312 TFLOP/s and HBM2e bandwidth is roughly 2 TB/s, giving a ridge point near 156 FLOP/byte. An operation must reuse each loaded byte ~156 times before it becomes compute-limited.

Why Matrix Multiplication Is Compute-Bound

A GEMM computing C = A × B with dimensions (M, K) × (K, N) does 2MKN FLOPs while reading MK + KN + MN elements. For large square matrices of size N:

AI ≈ 2N³ / 3N² = (2/3)N

AI grows linearly with N. At N = 512 in FP16 (2 bytes each), AI ≈ 341 FLOP/byte, well above the A100 ridge point. The tensor cores stay busy; memory is not the constraint. This is why increasing matrix size almost always improves GPU utilisation for GEMMs.

Contrast this with an elementwise addition C = A + B: two reads, one write, two FLOPs per element. AI ≈ 2/12 ≈ 0.17 FLOP/byte on FP32. The kernel is memory-bound by a factor of ~900 on the A100. More FP32 units would be wasted.

Profiling to Diagnose the Regime

Theory gives the ridge point; profiling confirms which side a real kernel lands on. NVIDIA Nsight Compute reports two metrics directly:

• sm__throughput.avg.pct_of_peak_sustained_elapsed: fraction of peak SM compute utilised
• l1tex__t_bytes.sum / dram__bytes.sum: memory traffic at different cache levels

A shortcut is to look at the roofline chart in Nsight Compute's "GPU Speed of Light" section. A kernel sitting far below the compute roofline and close to the memory bandwidth roofline is memory-bound. One crowding the compute ceiling is compute-bound.

For PyTorch users, torch.profiler with profile_memory=True and the Kineto backend surfaces per-kernel FLOP counts and memory read/write sizes, which feed directly into an AI calculation.

A practical heuristic: if reducing the output tensor's dtype from FP32 to FP16 cuts wall time nearly in half, the kernel is likely memory-bound (you halved the write traffic). If it barely changes, the kernel is compute-bound.

Optimisation Strategies Are Regime-Specific

Once you have a diagnosis, the toolkit diverges sharply.

Memory-bound kernels: the goal is to reduce bytes moved, not to do less arithmetic.

• Operator fusion: instead of writing an intermediate result back to HBM and re-reading it for the next kernel, keep it in registers or shared memory. PyTorch 2.x's torch.compile with the Inductor backend performs this automatically for pointwise chains.
• Kernel fusion (manual): FlashAttention rewrites the attention computation as a single tiled kernel so Q, K, V slices stay in SRAM. The FLOP count is identical to naive attention; the HBM traffic is dramatically lower.
• Mixed precision: halving element size halves bandwidth demand for the same computation.
• Reducing unnecessary reads: batching small operations that touch the same tensor avoids re-loading it.

Compute-bound kernels: the goal is to maximise ALU utilisation and hide latency.

• Tensor Core alignment: dimensions divisible by 16 (FP16) or 8 (TF32) map efficiently onto tensor core tiles. Odd dimensions cause padding waste.
• Occupancy tuning: enough thread blocks to fill all SMs, with enough registers and shared memory per block that warps can be in-flight to hide arithmetic latency.
• Reduced precision: FP16 or BF16 doubles the tensor-core FLOP/s versus FP32 on Ampere/Hopper. INT8 doubles again.
• Speculative decoding and batching: for inference, batching requests increases GEMM M dimensions, shifting the weight-loading cost over more output tokens and nudging the regime toward compute-bound.

When It Falls Down

The ridge point shifts with cache reuse. The Roofline model assumes memory traffic hits DRAM. If a kernel fits its working set in L2 (40 MB on A100) or L1/shared memory (~164 KB per SM), the effective bandwidth is 5-10x higher and the ridge point shifts right. A kernel classified as memory-bound against DRAM bandwidth may be compute-bound against L2 bandwidth. Nsight Compute's roofline supports separate lines for each memory level; use them.

Batching changes the classification. A single-batch decode pass through a transformer layer loads weight matrices once to produce one token. AI is tiny; the operation is memory-bound (the "batch size 1 inference problem"). At batch size 512, the same weights are reused 512 times, AI climbs, and the kernel becomes compute-bound. This is why LLM inference serving systems obsess over batching.

Fusion can overshoot. Fusing too many elementwise ops into one kernel can exhaust shared memory or register file, reducing occupancy below what is needed to hide latency. The fused kernel may end up slower than two separate unfused ones.

Quantisation is not always bandwidth-saving. INT4 weights halve the bandwidth of a FP8 load, but dequantisation adds FLOPs. If the kernel was already compute-bound, adding dequantisation FLOPs hurts; if it was memory-bound, the bandwidth saving dominates and total time drops.

AI calculated from source code can be misleading. Compilers re-order, vectorise, and sometimes re-read data due to register spilling. Profile-guided AI from Nsight is more reliable than hand-counted FLOPs.

Further Reading

• GPU Performance Background - NVIDIA Deep Learning Performance Guide
• Matrix Multiplication Background - NVIDIA Deep Learning Performance Guide
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., 2022)