Profiling GPU Workloads

A training step for a 7B-parameter model may spend 30% of its wall-clock time waiting on memory transfers that could be hidden with better tiling - but only if you can see the transfers at all. Without a profiler, every optimisation is archaeology: you make a change, time the whole run, and wonder whether the 2% speedup was real or noise. With a profiler you get a timeline that shows, to the microsecond, which kernel ran, how long it stalled on L2 misses, and whether the GPU was idle while the CPU was still queuing the next batch.

This concept explains how GPU profiling works, which tools to reach for at each layer of investigation, and which metrics actually predict whether a kernel is bottlenecked on compute or on memory.

The Two Bottleneck Categories

Every GPU kernel is either compute-bound or memory-bound. Understanding which one applies is the first job of any profiling session.

A kernel is compute-bound when arithmetic throughput (measured in TFLOP/s) is the limiting resource. Tensor-core GEMM on large matrices is the canonical case. A kernel is memory-bound when the rate at which data moves between DRAM and the SM register file is the constraint. Element-wise activations, layer normalisation, and many attention variants fall here.

The roofline model makes this crisp. Define arithmetic intensity as:

I = FLOPs / bytes_transferred

If I exceeds the machine's ridge point (peak FLOPs / peak memory bandwidth), the kernel is compute-bound; below it, memory-bound. An A100 SXM4 has a ridge point around 300 FLOP/byte for FP16 tensor-core operations. A softmax over a [batch, seq_len] tensor rarely exceeds 10 FLOP/byte, so it is firmly memory-bound regardless of how you schedule threads.

Knowing which regime you are in tells you where optimisation effort belongs: a memory-bound kernel gets faster from better tiling, coalescing, and fusion; a compute-bound kernel needs better tensor-core utilisation and register occupancy.

The Tool Stack

Profiling GPU workloads involves three layers of tooling, each answering a different question.

System-level: Nsight Systems (nsys)

Nsight Systems captures a timeline of everything: CPU threads, CUDA API calls, kernel launches, PCIe transfers, NCCL collectives, and NVLink traffic. It adds almost no overhead because it uses hardware performance counters and OS-level tracing. Use it first to find the bottleneck region before drilling deeper.

nsys profile --trace=cuda,nvtx,osrt \
             --output=profile_run \
             python train.py

The resulting .nsys-rep file opens in the Nsight Systems GUI. The timeline view immediately shows whether kernels overlap with data transfers (they should), whether there are long CPU gaps between kernel launches (a sign of Python overhead or synchronisation), and whether NVLink is saturating during an allreduce.

Kernel-level: Nsight Compute (ncu)

Once Nsight Systems has identified the hot kernels, Nsight Compute re-runs each one with full hardware counters to answer why it is slow. It reports achieved vs. theoretical occupancy, SM active cycles, memory throughput, cache hit rates, warp stall reasons, and whether tensor cores are actually being used.

ncu --set full \
    --kernel-name regex:attention \
    --launch-count 3 \
    python train.py

Key metrics to inspect:

High long_scoreboard stall percentage is the fingerprint of a memory-bound kernel. High no_eligible stall percentage means the warp scheduler has nothing to issue - a compute-bound kernel with insufficient parallelism.

Framework-level: torch.profiler

For PyTorch workloads, torch.profiler wraps CUDA events and the Kineto profiling library to produce Chrome-trace-compatible output viewable in chrome://tracing or TensorBoard. It is the right first step when you do not want to leave Python.

import torch
from torch.profiler import profile, ProfilerActivity, record_function

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    record_shapes=True,
    with_stack=True,
) as prof:
    with record_function("forward_pass"):
        output = model(batch)
        loss = criterion(output, targets)
        loss.backward()

print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
prof.export_chrome_trace("trace.json")

key_averages() groups events by operator name and shows CPU time, CUDA time, and memory allocated. This quickly surfaces whether a single operator (say, aten::_scaled_dot_product_flash_attention) dominates, or whether overhead is spread across many small kernels - each needing a different fix.

Reading the Numbers: A Worked Diagnosis

Suppose ncu reports that an attention kernel achieves 28% of peak memory bandwidth. That single number rules out the explanation "we are compute-bound" and points to three likely causes:

1. Uncoalesced global loads. Threads in a warp are accessing non-contiguous addresses. The fix is to restructure the data layout or use shared memory staging.
2. Low occupancy. Few warps are active per SM, so memory latency cannot be hidden. Check whether register spilling or shared-memory pressure is capping active warps.
3. Redundant loads. The same values are being reloaded from DRAM on each pass because they were never cached in shared memory. Tiling (loading a tile into SRAM, doing all the work on it, moving to the next tile) is the standard cure.

FlashAttention addresses all three for the attention operator: it tiles the K and V matrices so they fit in SRAM, computes the online softmax without materialising the full attention matrix, and achieves 60-70% of peak A100 HBM bandwidth on typical sequence lengths.

Annotating Your Code for Useful Profiles

Raw profiles of un-annotated code are hard to read because CUDA kernel names are auto-generated and bear no resemblance to the model operation that launched them. Two habits help.

NVTX ranges (used by both Nsys and ncu) let you label regions of code so they appear as coloured bands in the timeline:

import torch.cuda.nvtx as nvtx

nvtx.range_push("encoder_layer_3")
x = encoder_layer(x)
nvtx.range_pop()

record_function contexts in torch.profiler achieve the same thing at the framework level and appear in Chrome traces without needing NVTX.

In distributed training, annotating allreduce boundaries is especially valuable: it lets you verify that gradient compression or bucket coalescing is working as intended, rather than assuming it is.

When It Falls Down

Profiling tools have real limits that can mislead you if you forget them.

Heisenberg effect on small kernels. Nsight Compute replays kernels multiple times to collect all counter groups. For kernels shorter than about 10 microseconds, the replay overhead can exceed the kernel itself and the reported numbers become unreliable. Nsight Systems event timestamps are more trustworthy for very short kernels.

Counter multiplexing distorts roofline. Hardware has fewer counter registers than metrics. When ncu collects more counters than the hardware supports in a single pass, it runs the kernel multiple times across counter groups. If the kernel's behaviour is not perfectly reproducible (e.g., it uses atomics or its inputs change), the combined report will be inconsistent.

CUDA graph and torch.compile opacity. Once a model is compiled with torch.compile or captured into a CUDA graph, the kernel names in the profile change (often to generic triton_ or cudagraph_ names) and the relationship between a Python line and a CUDA kernel is less direct. Use TORCH_COMPILE_DEBUG=1 and the Inductor trace files to map compiled kernels back to their source operations.

Multi-GPU masking. On an 8-GPU node, profiling GPU 0 alone may give a misleading picture if allreduce collectives on other GPUs are creating backpressure. Profile all ranks simultaneously with nsys profile --mpi-impl=openmpi or equivalent, and look at collective timelines together.

Profiling != production behaviour. torch.profiler inserts synchronisation points that flush the CUDA queue between operator calls. This eliminates kernel-kernel overlap that exists in production, so absolute timings from torch.profiler are always pessimistic. Use Nsight Systems to see actual kernel overlap.

Further Reading

• NVIDIA Nsight Compute CLI Documentation - full reference for ncu flags, metrics, and counter groups.
• NVIDIA Nsight Systems User Guide - system-level timeline profiling, NVTX integration, and MPI support.
• PyTorch Profiler API Reference - torch.profiler.profile, ProfilerActivity, and Chrome-trace export.
• CUDA C++ Best Practices Guide - bandwidth measurement methodology, the APOD cycle, and occupancy analysis.