Tensors & Neural Plumbing
Shapes, matmuls, forward and backward passes, parameter counts, memory footprints.
18concepts
211flashcards
138minutes of reading
- 01 Checkpoint Formats and State Dicts A checkpoint is a dictionary from parameter names to tensors plus a pile of implicit assumptions about dtype, sharding, and key naming, and every one of those assumptions is somewhere a load goes wrong.
- 02 Counting Transformer Parameters A transformer's parameter count is not a mysterious headline number, it is the sum of a handful of matrix shapes multiplied out, and knowing the formula lets you sanity-check any model card in seconds.
- 03 Einsum and Tensor Contractions Einstein summation notation expresses matmuls, batched matmuls, and attention itself as one uniform pattern, and reading it fluently is the fastest way to understand what a line of unfamiliar model code actually computes.
- 04 Forward-Mode and Reverse-Mode Autodiff Backpropagation is one of two ways to apply the chain rule mechanically, and the choice between them is decided by a single number, the ratio of inputs to outputs, which is why training uses reverse mode and Hessian-vector products use both.
- 05 Padding, Masks, and Variable-Length Batching Real batches contain sequences of different lengths, and the three ways of handling that difference, padding, packing, and varlen kernels, have wildly different costs and each has a signature bug.
- 06 Tensor Memory Layout and Contiguity A tensor is a pointer, a shape, and a stride tuple; transposes and slices change only the metadata, which is why some reshapes are free, some silently copy gigabytes, and a permute in the wrong place can halve your throughput.
- 07 The Backward Pass and Gradient Flow Backpropagation through a transformer is the forward pass run in reverse with the chain rule attached; seeing which operations preserve gradient magnitude and which shrink it explains why architecture choices exist.
- 08 The Forward Pass End to End Trace one token's numbers from embedding table to output logits and every "mysterious" transformer component turns out to be a shape-preserving or shape-mixing step in a fixed pipeline.
- 09 Views, Aliasing, and In-Place Operations A transpose returns a new tensor object that shares its old storage, which is why in-place edits leak across variables you thought were separate and why autograd raises "a variable needed for gradient computation has been modified".