Tensors & Neural Plumbing
Shapes, matmuls, forward and backward passes, parameter counts, memory footprints.
10concepts
42flashcards
77minutes of reading
- 01 Counting Transformer Parameters A transformer's parameter count is not a mysterious headline number, it is the sum of a handful of matrix shapes multiplied out, and knowing the formula lets you sanity-check any model card in seconds.
- 02 Einsum and Tensor Contractions Einstein summation notation expresses matmuls, batched matmuls, and attention itself as one uniform pattern, and reading it fluently is the fastest way to understand what a line of unfamiliar model code actually computes.
- 03 The Backward Pass and Gradient Flow Backpropagation through a transformer is the forward pass run in reverse with the chain rule attached; seeing which operations preserve gradient magnitude and which shrink it explains why architecture choices exist.
- 04 The Forward Pass End to End Trace one token's numbers from embedding table to output logits and every "mysterious" transformer component turns out to be a shape-preserving or shape-mixing step in a fixed pipeline.