Tensors & Neural Plumbing
Shapes, matmuls, forward and backward passes, parameter counts, memory footprints.
10concepts
42flashcards
77minutes of reading
- 01 Broadcasting and Vectorisation Broadcasting is the rule that lets a bias vector add to every row of a batch without an explicit loop, and understanding its shape-matching logic prevents the class of bugs that produce wrong answers without an error.
- 02 Matrix Multiplication: The Core Operation Nearly every FLOP a transformer spends is a matrix multiplication; understanding its shape rule and its cost is the single most load-bearing piece of maths in deep learning.
- 03 Tensors, Shapes, and Batching Every number a transformer touches lives inside a tensor with a fixed shape; learning to read and predict those shapes is the fastest way to stop being confused by model code.
- 04 Why Non-Linearity Matters Stack any number of linear layers and the result is still one linear layer; the humble activation function is the only thing standing between a transformer and a glorified matrix multiplication.
- 05 Counting Transformer Parameters A transformer's parameter count is not a mysterious headline number, it is the sum of a handful of matrix shapes multiplied out, and knowing the formula lets you sanity-check any model card in seconds.
- 06 Einsum and Tensor Contractions Einstein summation notation expresses matmuls, batched matmuls, and attention itself as one uniform pattern, and reading it fluently is the fastest way to understand what a line of unfamiliar model code actually computes.
- 07 The Backward Pass and Gradient Flow Backpropagation through a transformer is the forward pass run in reverse with the chain rule attached; seeing which operations preserve gradient magnitude and which shrink it explains why architecture choices exist.
- 08 The Forward Pass End to End Trace one token's numbers from embedding table to output logits and every "mysterious" transformer component turns out to be a shape-preserving or shape-mixing step in a fixed pipeline.
- 09 Signal Propagation and Initialisation How a network's weights are initialised decides, before a single gradient step, whether activations and gradients stay in a trainable range or collapse to zero or explode across depth.
- 10 Training Memory Footprint A model's weights are the smallest part of its training memory bill; optimiser state, gradients, and activations usually cost several times more, and knowing the breakdown explains why training needs far more memory than inference.