Tensors & Neural Plumbing
Shapes, matmuls, forward and backward passes, parameter counts, memory footprints.
18concepts
211flashcards
138minutes of reading
- 01 Broadcasting and Vectorisation Broadcasting is the rule that lets a bias vector add to every row of a batch without an explicit loop, and understanding its shape-matching logic prevents the class of bugs that produce wrong answers without an error.
- 02 Matrix Multiplication: The Core Operation Nearly every FLOP a transformer spends is a matrix multiplication; understanding its shape rule and its cost is the single most load-bearing piece of maths in deep learning.
- 03 Tensors, Shapes, and Batching Every number a transformer touches lives inside a tensor with a fixed shape; learning to read and predict those shapes is the fastest way to stop being confused by model code.
- 04 Why Non-Linearity Matters Stack any number of linear layers and the result is still one linear layer; the humble activation function is the only thing standing between a transformer and a glorified matrix multiplication.
- 05 Checkpoint Formats and State Dicts A checkpoint is a dictionary from parameter names to tensors plus a pile of implicit assumptions about dtype, sharding, and key naming, and every one of those assumptions is somewhere a load goes wrong.
- 06 Counting Transformer Parameters A transformer's parameter count is not a mysterious headline number, it is the sum of a handful of matrix shapes multiplied out, and knowing the formula lets you sanity-check any model card in seconds.
- 07 Einsum and Tensor Contractions Einstein summation notation expresses matmuls, batched matmuls, and attention itself as one uniform pattern, and reading it fluently is the fastest way to understand what a line of unfamiliar model code actually computes.
- 08 Forward-Mode and Reverse-Mode Autodiff Backpropagation is one of two ways to apply the chain rule mechanically, and the choice between them is decided by a single number, the ratio of inputs to outputs, which is why training uses reverse mode and Hessian-vector products use both.
- 09 Padding, Masks, and Variable-Length Batching Real batches contain sequences of different lengths, and the three ways of handling that difference, padding, packing, and varlen kernels, have wildly different costs and each has a signature bug.
- 10 Tensor Memory Layout and Contiguity A tensor is a pointer, a shape, and a stride tuple; transposes and slices change only the metadata, which is why some reshapes are free, some silently copy gigabytes, and a permute in the wrong place can halve your throughput.
- 11 The Backward Pass and Gradient Flow Backpropagation through a transformer is the forward pass run in reverse with the chain rule attached; seeing which operations preserve gradient magnitude and which shrink it explains why architecture choices exist.
- 12 The Forward Pass End to End Trace one token's numbers from embedding table to output logits and every "mysterious" transformer component turns out to be a shape-preserving or shape-mixing step in a fixed pipeline.
- 13 Views, Aliasing, and In-Place Operations A transpose returns a new tensor object that shares its old storage, which is why in-place edits leak across variables you thought were separate and why autograd raises "a variable needed for gradient computation has been modified".
- 14 Accumulation Precision and Mixed-Dtype Plumbing Storing weights in 16 bits is safe because almost nothing is actually computed in 16 bits — matmuls accumulate in fp32, reductions run in fp32, and the master copy is fp32, and every documented failure of low-precision training is one of these rules being broken.
- 15 Batch Invariance and Numerical Reproducibility Temperature zero does not give the same answer twice, and the reason is neither random seeds nor GPU atomics; it is that reduction kernels change their split strategy with batch shape, so your logits depend on which other requests happened to be in flight.
- 16 Signal Propagation and Initialisation How a network's weights are initialised decides, before a single gradient step, whether activations and gradients stay in a trainable range or collapse to zero or explode across depth.
- 17 Training Memory Footprint A model's weights are the smallest part of its training memory bill; optimiser state, gradients, and activations usually cost several times more, and knowing the breakdown explains why training needs far more memory than inference.
- 18 muP and Hyperparameter Transfer Under standard parametrisation the best learning rate drifts as a model gets wider, so every scale-up is a fresh search; muP rescales initialisation, learning rates, and multipliers per layer so the optimum stays put and can be tuned on a small proxy model.