Kernels & Compilers
CUDA, Triton, fusion, tiling, torch.compile, CUDA graphs and roofline-guided optimisation.
20concepts
140flashcards
159minutes of reading
- 01 What a CUDA Kernel Is A CUDA kernel is a C++ function that runs simultaneously on thousands of GPU threads, each identified by a coordinate in a structured grid, and understanding this execution model is the prerequisite for reasoning about throughput in any deep-learning workload.
- 02 Graph Capture and CUDA Graphs CUDA Graphs record a sequence of GPU operations as a reusable execution graph, eliminating per-kernel CPU launch overhead and enabling significant throughput gains for workloads with static shapes and control flow.
- 03 Kernel Fusion Kernel fusion eliminates redundant memory round-trips by merging multiple GPU operations into a single kernel launch, turning memory-bandwidth bottlenecks into throughput wins.
- 04 Memory Coalescing Memory coalescing is the hardware mechanism by which a GPU groups multiple thread memory requests into a single wide transaction, and writing kernels that exploit it is often the single largest lever on throughput.
- 05 Operator Lowering and IRs Operator lowering is the process of progressively translating high-level tensor operations through a sequence of intermediate representations until hardware-executable instructions are produced.
- 06 Profiling GPU Workloads Profiling a GPU workload means measuring where time and memory bandwidth actually go, so that optimisation effort lands on the real bottleneck rather than a guess.
- 07 Shared Memory and Tiling Shared memory is a programmer-controlled on-chip SRAM that lets a thread block reuse data without re-fetching it from global memory, and tiling is the technique that makes that reuse systematic.
- 08 The CUDA Programming Model CUDA organises GPU execution into a three-level hierarchy of grids, blocks, and threads, and every performance decision traces back to how well that hierarchy is exploited.
- 09 Triton: Python-Level GPU Kernels Triton lets you write GPU kernels in Python by operating on tiles of data rather than individual threads, and its compiler handles shared-memory management, coalescing, and vectorisation automatically.
- 10 When Not to Write a Custom Kernel Writing a CUDA kernel is expensive to maintain and easy to get wrong; this concept maps the decision boundary between writing one and leaning on existing compilers and libraries.
- 11 Writing a Fused Softmax A fused softmax kernel collapses three separate memory-bound passes over a matrix row into one, cutting HBM traffic by roughly 4x and turning a memory-bound operation into a compute-limited one.
- 12 XLA and Just-In-Time Compilation XLA compiles a whole computation graph into fused, hardware-specific kernels at runtime, trading a one-time compilation cost for sustained throughput gains across GPUs and TPUs.
- 13 torch.compile and TorchInductor torch.compile traces PyTorch graphs at runtime via TorchDynamo, then lowers them through TorchInductor to fused Triton or C++ kernels, delivering 20-36% throughput gains with no model rewrites.
- 14 Autotuning GPU Kernels Autotuning systematically searches a discrete configuration space of tile sizes, warp counts, and pipeline stages to find the fastest kernel for a given GPU and problem shape, replacing manual heuristics with empirical benchmarking.
- 15 Custom Kernels for Mixture-of-Experts MoE models break the dense-GEMM assumption that GPU libraries are optimised for, so efficient inference requires custom grouped-GEMM and block-sparse kernels that handle variable-length expert batches without padding or token dropping.
- 16 Mixed-Precision Kernels Mixed-precision kernels reduce memory bandwidth and arithmetic cost by storing and computing in lower-precision formats while selectively preserving full precision where numerical stability demands it.
- 17 Paged Attention as a Memory Manager PagedAttention borrows the OS virtual-memory paging model to eliminate KV-cache fragmentation, letting a single GPU serve far more concurrent requests than contiguous allocation allows.
- 18 Quantised GEMM Kernels Quantised GEMM kernels replace 16-bit or 32-bit matrix multiplications with 8-bit or 4-bit integer arithmetic, cutting memory bandwidth and compute cost while preserving model accuracy through careful scaling and outlier handling.
- 19 Roofline-Guided Kernel Optimisation The roofline model maps a kernel's arithmetic intensity against hardware ceilings to diagnose whether compute or memory bandwidth is the binding constraint, and directs every subsequent optimisation decision.
- 20 Why FlashAttention Is a Kernel Story FlashAttention achieves its speedups not by reducing FLOPs but by restructuring the attention computation into a single tiled CUDA kernel that fits working data in on-chip SRAM, eliminating the dominant cost of round-tripping through GPU HBM.