Tensors & Neural Plumbing
Shapes, matmuls, forward and backward passes, parameter counts, memory footprints.
18concepts
211flashcards
138minutes of reading
- 01 Accumulation Precision and Mixed-Dtype Plumbing Storing weights in 16 bits is safe because almost nothing is actually computed in 16 bits — matmuls accumulate in fp32, reductions run in fp32, and the master copy is fp32, and every documented failure of low-precision training is one of these rules being broken.
- 02 Batch Invariance and Numerical Reproducibility Temperature zero does not give the same answer twice, and the reason is neither random seeds nor GPU atomics; it is that reduction kernels change their split strategy with batch shape, so your logits depend on which other requests happened to be in flight.
- 03 Signal Propagation and Initialisation How a network's weights are initialised decides, before a single gradient step, whether activations and gradients stay in a trainable range or collapse to zero or explode across depth.
- 04 Training Memory Footprint A model's weights are the smallest part of its training memory bill; optimiser state, gradients, and activations usually cost several times more, and knowing the breakdown explains why training needs far more memory than inference.
- 05 muP and Hyperparameter Transfer Under standard parametrisation the best learning rate drifts as a model gets wider, so every scale-up is a fresh search; muP rescales initialisation, learning rates, and multipliers per layer so the optimum stays put and can be tuned on a small proxy model.