Concept library
446 concepts across 8 domains and 36 tracks. Each track is a coherent sequence — read it top to bottom or dip in wherever the gap is.
05
Inference, Systems & Hardware
Where the model meets the silicon, the memory bus and the latency budget.
4tracks
55concepts
379cards
7.4hreading
Inference Optimisation KV cache, FlashAttention, speculative decoding, quantisation and continuous batching. 7 concepts · 33 cards
- 01 KV Cache Why decoder inference is quadratic without a KV cache and linear with one, and why managing that cache is now the dominant memory problem in LLM serving.
- 02 Quantisation - INT8, INT4, FP8 How to cut weight and activation precision below 16 bits without wrecking quality, and which scheme to pick for which deployment.
- 03 vLLM and Continuous Batching Why static batching wastes most of your GPU on variable-length workloads, and how iteration-level scheduling combined with PagedAttention raises throughput by an order of magnitude.
- 04 FlashAttention An IO-aware attention kernel that is both faster and lower-memory than the textbook implementation by tiling computation to keep activations in SRAM.
- 05 Mixture-of-Experts Inference Why serving MoE models is harder than serving dense models of equivalent quality, and how DeepSeek and Mistral made it work in production.
- 06 Paged Attention and the KV Memory Manager How vLLM's PagedAttention treats the KV cache like OS virtual memory, cutting fragmentation waste from 60-80% to under 4% and roughly doubling to quadrupling serving throughput.
- 07 Speculative Decoding Use a small draft model to propose tokens that a large verifier accepts or rejects in parallel, giving lossless 2-3x latency wins on autoregressive generation.
Accelerator Architecture The memory wall, roofline analysis, GPU execution model, interconnects and systolic arrays. 20 concepts · 140 cards
- 01 Collective Communication Primitives The six core multi-GPU communication patterns (broadcast, reduce, all-reduce, all-gather, reduce-scatter, all-to-all) determine whether a distributed training job spends most of its time computing or waiting on the wire.
- 02 Compute-Bound vs Memory-Bound Kernels A kernel's performance ceiling is determined by whether FLOPs or memory bandwidth runs out first, and misidentifying this wastes orders-of-magnitude optimisation effort.
- 03 Floating-Point Formats for ML ML accelerators expose a menu of floating-point formats that trade numerical range and precision for throughput and memory bandwidth; choosing the wrong one silently degrades accuracy or leaves peak FLOPS on the table.
- 04 HBM Bandwidth and Capacity High Bandwidth Memory sets a hard ceiling on how fast a GPU can feed its compute units, and most LLM operations live squarely against that ceiling.
- 05 InfiniBand and Inter-Node Networking InfiniBand provides low-latency, high-bandwidth RDMA links between GPU nodes, and understanding its topology and collective communication patterns is essential for diagnosing and eliminating the network bottleneck in large-scale training.
- 06 KV-Cache Memory and Bandwidth The key-value cache trades GPU memory capacity for inference speed, and understanding how that trade interacts with memory bandwidth is what separates fast serving systems from slow ones.
- 07 NVLink and Intra-Node Interconnect NVLink is NVIDIA's proprietary GPU-to-GPU interconnect that delivers up to 900 GB/s aggregate bandwidth on H100, replacing PCIe as the bottleneck in multi-GPU training by making all-reduce and tensor parallelism far cheaper.
- 08 Occupancy and Latency Hiding GPU occupancy measures how many warps are resident on a streaming multiprocessor relative to its hardware maximum, and high occupancy is the primary mechanism by which the GPU hides memory and arithmetic latency to sustain throughput.
- 09 Power, Thermals, and Clock Throttling GPU accelerators operate under hard power and thermal budgets that silently reduce clock speeds mid-workload, making sustained throughput lower than peak spec sheets advertise.
- 10 Prefill vs Decode LLM inference splits into two hardware-distinct phases - a compute-bound prefill that processes all prompt tokens in parallel, and a memory-bandwidth-bound decode that generates tokens one at a time, each with fundamentally different bottlenecks on the same GPU.
- 11 Reading an Accelerator Datasheet A datasheet number means nothing without the four unit-aware ratios that reveal whether your workload will actually be compute-bound or memory-bound on that chip.
- 12 TPU Systolic Arrays A systolic array is a grid of multiply-accumulate units wired to pass partial sums directly between neighbours, letting Google's TPU sustain 92 TOPS on matrix multiplication without repeatedly hitting off-chip memory.
- 13 Tensor Cores and Matrix Engines Tensor Cores are specialised matrix-multiply-accumulate units on modern GPUs that deliver peak FLOP/s only when operand shapes and numeric formats are chosen correctly.
- 14 The FLOPs of a Transformer Forward Pass A systematic derivation of how many floating-point operations a single transformer forward pass costs, and why that number dictates hardware choice, batch strategy, and scaling decisions.
- 15 The GPU Execution Model GPUs execute thousands of threads in lockstep groups called warps; understanding that hierarchy and where threads stall is the single most important mental model for writing fast GPU code.
- 16 The GPU Memory Hierarchy A GPU's memory is a multi-tier hierarchy where bandwidth drops and latency rises by orders of magnitude as you move outward from registers to HBM, and the speed of your kernel is almost always determined by which tier bottlenecks it.
- 17 The Memory Wall and Arithmetic Intensity Arithmetic intensity determines whether a GPU kernel is memory-bound or compute-bound, and almost every LLM inference operation sits on the wrong side of that line.
- 18 The Roofline Model The Roofline Model bounds attainable hardware performance using two ceilings - peak compute throughput and peak memory bandwidth - letting you diagnose whether a kernel wastes silicon or wasits time waiting for data.
- 19 Why GEMMs Dominate Almost every compute-heavy operation in a neural network reduces to a matrix multiply, which is why hardware and compilers optimise almost exclusively for GEMM throughput.
- 20 Hardware Cost of Mixture-of-Experts Sparse MoE models reduce FLOPs per token but introduce all-to-all communication, load-imbalance penalties, and memory pressure that can erase those savings unless the system is carefully co-designed.
Kernels & Compilers CUDA, Triton, fusion, tiling, torch.compile, CUDA graphs and roofline-guided optimisation. 20 concepts · 140 cards
- 01 What a CUDA Kernel Is A CUDA kernel is a C++ function that runs simultaneously on thousands of GPU threads, each identified by a coordinate in a structured grid, and understanding this execution model is the prerequisite for reasoning about throughput in any deep-learning workload.
- 02 Graph Capture and CUDA Graphs CUDA Graphs record a sequence of GPU operations as a reusable execution graph, eliminating per-kernel CPU launch overhead and enabling significant throughput gains for workloads with static shapes and control flow.
- 03 Kernel Fusion Kernel fusion eliminates redundant memory round-trips by merging multiple GPU operations into a single kernel launch, turning memory-bandwidth bottlenecks into throughput wins.
- 04 Memory Coalescing Memory coalescing is the hardware mechanism by which a GPU groups multiple thread memory requests into a single wide transaction, and writing kernels that exploit it is often the single largest lever on throughput.
- 05 Operator Lowering and IRs Operator lowering is the process of progressively translating high-level tensor operations through a sequence of intermediate representations until hardware-executable instructions are produced.
- 06 Profiling GPU Workloads Profiling a GPU workload means measuring where time and memory bandwidth actually go, so that optimisation effort lands on the real bottleneck rather than a guess.
- 07 Shared Memory and Tiling Shared memory is a programmer-controlled on-chip SRAM that lets a thread block reuse data without re-fetching it from global memory, and tiling is the technique that makes that reuse systematic.
- 08 The CUDA Programming Model CUDA organises GPU execution into a three-level hierarchy of grids, blocks, and threads, and every performance decision traces back to how well that hierarchy is exploited.
- 09 Triton: Python-Level GPU Kernels Triton lets you write GPU kernels in Python by operating on tiles of data rather than individual threads, and its compiler handles shared-memory management, coalescing, and vectorisation automatically.
- 10 When Not to Write a Custom Kernel Writing a CUDA kernel is expensive to maintain and easy to get wrong; this concept maps the decision boundary between writing one and leaning on existing compilers and libraries.
- 11 Writing a Fused Softmax A fused softmax kernel collapses three separate memory-bound passes over a matrix row into one, cutting HBM traffic by roughly 4x and turning a memory-bound operation into a compute-limited one.
- 12 XLA and Just-In-Time Compilation XLA compiles a whole computation graph into fused, hardware-specific kernels at runtime, trading a one-time compilation cost for sustained throughput gains across GPUs and TPUs.
- 13 torch.compile and TorchInductor torch.compile traces PyTorch graphs at runtime via TorchDynamo, then lowers them through TorchInductor to fused Triton or C++ kernels, delivering 20-36% throughput gains with no model rewrites.
- 14 Autotuning GPU Kernels Autotuning systematically searches a discrete configuration space of tile sizes, warp counts, and pipeline stages to find the fastest kernel for a given GPU and problem shape, replacing manual heuristics with empirical benchmarking.
- 15 Custom Kernels for Mixture-of-Experts MoE models break the dense-GEMM assumption that GPU libraries are optimised for, so efficient inference requires custom grouped-GEMM and block-sparse kernels that handle variable-length expert batches without padding or token dropping.
- 16 Mixed-Precision Kernels Mixed-precision kernels reduce memory bandwidth and arithmetic cost by storing and computing in lower-precision formats while selectively preserving full precision where numerical stability demands it.
- 17 Paged Attention as a Memory Manager PagedAttention borrows the OS virtual-memory paging model to eliminate KV-cache fragmentation, letting a single GPU serve far more concurrent requests than contiguous allocation allows.
- 18 Quantised GEMM Kernels Quantised GEMM kernels replace 16-bit or 32-bit matrix multiplications with 8-bit or 4-bit integer arithmetic, cutting memory bandwidth and compute cost while preserving model accuracy through careful scaling and outlier handling.
- 19 Roofline-Guided Kernel Optimisation The roofline model maps a kernel's arithmetic intensity against hardware ceilings to diagnose whether compute or memory bandwidth is the binding constraint, and directs every subsequent optimisation decision.
- 20 Why FlashAttention Is a Kernel Story FlashAttention achieves its speedups not by reducing FLOPs but by restructuring the attention computation into a single tiled CUDA kernel that fits working data in on-chip SRAM, eliminating the dominant cost of round-tripping through GPU HBM.
Serving Systems Prompt caching, gateways and routing, token accounting, and multi-tenant isolation. 8 concepts · 66 cards
- 01 Autoscaling and Cold Starts in LLM Serving Why GPU utilisation is a useless autoscaling signal for LLM servers, what a cold start actually costs, and how to scale a fleet whose new replicas take minutes to become useful.
- 02 LLM Gateways and Routing Why every serious LLM deployment ends up behind a gateway, and how to choose between LiteLLM, Portkey, OpenRouter, and rolling your own.
- 03 Prompt Caching Infrastructure How Anthropic, OpenAI, and vLLM let you reuse the KV cache of repeated prefixes, what the cache key actually is, and the patterns that turn cache hit rate into a real bill reduction.
- 04 Serving SLOs: TTFT, TPOT and Goodput Why tokens per second is the wrong number to optimise, how TTFT and TPOT split the latency budget, and what goodput measures that throughput hides.
- 05 Token Accounting, Billing, and Quotas Why a single token counter is not enough, how to attribute spend across users and features without losing your mind, and the patterns that prevent one bad actor from spending the whole month's budget on a Tuesday afternoon.
- 06 Disaggregated Prefill and Decode Serving Why prefill and decode want opposite hardware and parallelism, how splitting them across separate GPU pools raises goodput, and what the KV cache transfer costs.
- 07 Multi-Tenant Serving and Isolation Serving many tenants from one model is cheap and easy; giving each tenant their own fine-tune is expensive and hard. S-LoRA and per-request LoRA serving collapse the trade-off, but only for tenants who can share a base model.
- 08 Prefix-Aware Routing and KV Cache Reuse Why load-balancing LLM requests round-robin throws away computed KV cache, and how routing on prompt prefix turns a fleet's caches into a shared asset.