Deep Learning Building Blocks
Convolutions, recurrence, normalisation, activations, optimisers and regularisation.
8concepts
37flashcards
62minutes of reading
- 01 Dropout and Modern Regularisation Why dropout was the dominant regulariser for a decade and why modern LLM training mostly skips it in favour of letting data do the work.
- 02 Activation Functions Why depth is meaningless without a nonlinearity, and how the field moved from saturating sigmoids through ReLU and GELU to the gated SwiGLU that sits in almost every modern transformer feed-forward block.
- 03 Backpropagation and Automatic Differentiation How reverse-mode autodiff turns the chain rule into an efficient gradient algorithm, and the design choices PyTorch and JAX make to implement it at scale.
- 04 Convolutional Neural Networks Why weight sharing and local receptive fields make CNNs the right inductive bias for images, and where ViTs took over.
- 05 Normalisation: BatchNorm, LayerNorm, RMSNorm Why normalisation accelerates training, why transformers use LayerNorm instead of BatchNorm, and why RMSNorm is now the default in Llama-class models.
- 06 Optimisers: SGD, Adam, AdamW, Lion How the standard optimiser stack evolved from plain SGD through Adam to memory-cheaper variants like Lion and Muon, and which learning-rate schedules actually work at scale.
- 07 Recurrent Networks: RNN, LSTM, GRU How gating fixed the vanishing-gradient problem in RNNs, and why transformers displaced them everywhere except streaming and on-device workloads.
- 08 Residual Connections and Skip Paths Why adding the input back to a layer's output is what makes networks of dozens or hundreds of layers trainable, and how the same trick underpins every modern transformer.