Kernels & Compilers
CUDA, Triton, fusion, tiling, torch.compile, CUDA graphs and roofline-guided optimisation.
20concepts
140flashcards
159minutes of reading
- 01 Graph Capture and CUDA Graphs CUDA Graphs record a sequence of GPU operations as a reusable execution graph, eliminating per-kernel CPU launch overhead and enabling significant throughput gains for workloads with static shapes and control flow.
- 02 Kernel Fusion Kernel fusion eliminates redundant memory round-trips by merging multiple GPU operations into a single kernel launch, turning memory-bandwidth bottlenecks into throughput wins.
- 03 Memory Coalescing Memory coalescing is the hardware mechanism by which a GPU groups multiple thread memory requests into a single wide transaction, and writing kernels that exploit it is often the single largest lever on throughput.
- 04 Operator Lowering and IRs Operator lowering is the process of progressively translating high-level tensor operations through a sequence of intermediate representations until hardware-executable instructions are produced.
- 05 Profiling GPU Workloads Profiling a GPU workload means measuring where time and memory bandwidth actually go, so that optimisation effort lands on the real bottleneck rather than a guess.
- 06 Shared Memory and Tiling Shared memory is a programmer-controlled on-chip SRAM that lets a thread block reuse data without re-fetching it from global memory, and tiling is the technique that makes that reuse systematic.
- 07 The CUDA Programming Model CUDA organises GPU execution into a three-level hierarchy of grids, blocks, and threads, and every performance decision traces back to how well that hierarchy is exploited.
- 08 Triton: Python-Level GPU Kernels Triton lets you write GPU kernels in Python by operating on tiles of data rather than individual threads, and its compiler handles shared-memory management, coalescing, and vectorisation automatically.
- 09 When Not to Write a Custom Kernel Writing a CUDA kernel is expensive to maintain and easy to get wrong; this concept maps the decision boundary between writing one and leaning on existing compilers and libraries.
- 10 Writing a Fused Softmax A fused softmax kernel collapses three separate memory-bound passes over a matrix row into one, cutting HBM traffic by roughly 4x and turning a memory-bound operation into a compute-limited one.
- 11 XLA and Just-In-Time Compilation XLA compiles a whole computation graph into fused, hardware-specific kernels at runtime, trading a one-time compilation cost for sustained throughput gains across GPUs and TPUs.
- 12 torch.compile and TorchInductor torch.compile traces PyTorch graphs at runtime via TorchDynamo, then lowers them through TorchInductor to fused Triton or C++ kernels, delivering 20-36% throughput gains with no model rewrites.