Tensors & Neural Plumbing
Shapes, matmuls, forward and backward passes, parameter counts, memory footprints.
10concepts
42flashcards
77minutes of reading
- 01 Signal Propagation and Initialisation How a network's weights are initialised decides, before a single gradient step, whether activations and gradients stay in a trainable range or collapse to zero or explode across depth.
- 02 Training Memory Footprint A model's weights are the smallest part of its training memory bill; optimiser state, gradients, and activations usually cost several times more, and knowing the breakdown explains why training needs far more memory than inference.