Accelerator Architecture
The memory wall, roofline analysis, GPU execution model, interconnects and systolic arrays.
20concepts
140flashcards
158minutes of reading
- 01 Collective Communication Primitives The six core multi-GPU communication patterns (broadcast, reduce, all-reduce, all-gather, reduce-scatter, all-to-all) determine whether a distributed training job spends most of its time computing or waiting on the wire.
- 02 Compute-Bound vs Memory-Bound Kernels A kernel's performance ceiling is determined by whether FLOPs or memory bandwidth runs out first, and misidentifying this wastes orders-of-magnitude optimisation effort.
- 03 Floating-Point Formats for ML ML accelerators expose a menu of floating-point formats that trade numerical range and precision for throughput and memory bandwidth; choosing the wrong one silently degrades accuracy or leaves peak FLOPS on the table.
- 04 HBM Bandwidth and Capacity High Bandwidth Memory sets a hard ceiling on how fast a GPU can feed its compute units, and most LLM operations live squarely against that ceiling.
- 05 InfiniBand and Inter-Node Networking InfiniBand provides low-latency, high-bandwidth RDMA links between GPU nodes, and understanding its topology and collective communication patterns is essential for diagnosing and eliminating the network bottleneck in large-scale training.
- 06 KV-Cache Memory and Bandwidth The key-value cache trades GPU memory capacity for inference speed, and understanding how that trade interacts with memory bandwidth is what separates fast serving systems from slow ones.
- 07 NVLink and Intra-Node Interconnect NVLink is NVIDIA's proprietary GPU-to-GPU interconnect that delivers up to 900 GB/s aggregate bandwidth on H100, replacing PCIe as the bottleneck in multi-GPU training by making all-reduce and tensor parallelism far cheaper.
- 08 Occupancy and Latency Hiding GPU occupancy measures how many warps are resident on a streaming multiprocessor relative to its hardware maximum, and high occupancy is the primary mechanism by which the GPU hides memory and arithmetic latency to sustain throughput.
- 09 Power, Thermals, and Clock Throttling GPU accelerators operate under hard power and thermal budgets that silently reduce clock speeds mid-workload, making sustained throughput lower than peak spec sheets advertise.
- 10 Prefill vs Decode LLM inference splits into two hardware-distinct phases - a compute-bound prefill that processes all prompt tokens in parallel, and a memory-bandwidth-bound decode that generates tokens one at a time, each with fundamentally different bottlenecks on the same GPU.
- 11 Reading an Accelerator Datasheet A datasheet number means nothing without the four unit-aware ratios that reveal whether your workload will actually be compute-bound or memory-bound on that chip.
- 12 TPU Systolic Arrays A systolic array is a grid of multiply-accumulate units wired to pass partial sums directly between neighbours, letting Google's TPU sustain 92 TOPS on matrix multiplication without repeatedly hitting off-chip memory.
- 13 Tensor Cores and Matrix Engines Tensor Cores are specialised matrix-multiply-accumulate units on modern GPUs that deliver peak FLOP/s only when operand shapes and numeric formats are chosen correctly.
- 14 The FLOPs of a Transformer Forward Pass A systematic derivation of how many floating-point operations a single transformer forward pass costs, and why that number dictates hardware choice, batch strategy, and scaling decisions.
- 15 The GPU Execution Model GPUs execute thousands of threads in lockstep groups called warps; understanding that hierarchy and where threads stall is the single most important mental model for writing fast GPU code.
- 16 The GPU Memory Hierarchy A GPU's memory is a multi-tier hierarchy where bandwidth drops and latency rises by orders of magnitude as you move outward from registers to HBM, and the speed of your kernel is almost always determined by which tier bottlenecks it.
- 17 The Memory Wall and Arithmetic Intensity Arithmetic intensity determines whether a GPU kernel is memory-bound or compute-bound, and almost every LLM inference operation sits on the wrong side of that line.
- 18 The Roofline Model The Roofline Model bounds attainable hardware performance using two ceilings - peak compute throughput and peak memory bandwidth - letting you diagnose whether a kernel wastes silicon or wasits time waiting for data.
- 19 Why GEMMs Dominate Almost every compute-heavy operation in a neural network reduces to a matrix multiply, which is why hardware and compilers optimise almost exclusively for GEMM throughput.