Kernels & Compilers
CUDA, Triton, fusion, tiling, torch.compile, CUDA graphs and roofline-guided optimisation.
20concepts
140flashcards
159minutes of reading
- 01 Autotuning GPU Kernels Autotuning systematically searches a discrete configuration space of tile sizes, warp counts, and pipeline stages to find the fastest kernel for a given GPU and problem shape, replacing manual heuristics with empirical benchmarking.
- 02 Custom Kernels for Mixture-of-Experts MoE models break the dense-GEMM assumption that GPU libraries are optimised for, so efficient inference requires custom grouped-GEMM and block-sparse kernels that handle variable-length expert batches without padding or token dropping.
- 03 Mixed-Precision Kernels Mixed-precision kernels reduce memory bandwidth and arithmetic cost by storing and computing in lower-precision formats while selectively preserving full precision where numerical stability demands it.
- 04 Paged Attention as a Memory Manager PagedAttention borrows the OS virtual-memory paging model to eliminate KV-cache fragmentation, letting a single GPU serve far more concurrent requests than contiguous allocation allows.
- 05 Quantised GEMM Kernels Quantised GEMM kernels replace 16-bit or 32-bit matrix multiplications with 8-bit or 4-bit integer arithmetic, cutting memory bandwidth and compute cost while preserving model accuracy through careful scaling and outlier handling.
- 06 Roofline-Guided Kernel Optimisation The roofline model maps a kernel's arithmetic intensity against hardware ceilings to diagnose whether compute or memory bandwidth is the binding constraint, and directs every subsequent optimisation decision.
- 07 Why FlashAttention Is a Kernel Story FlashAttention achieves its speedups not by reducing FLOPs but by restructuring the attention computation into a single tiled CUDA kernel that fits working data in on-chip SRAM, eliminating the dominant cost of round-tripping through GPU HBM.