Deep Learning Building Blocks
Convolutions, recurrence, normalisation, activations, optimisers and regularisation.
13concepts
112flashcards
102minutes of reading
- 01 Dropout and Modern Regularisation Why dropout was the dominant regulariser for a decade and why modern LLM training mostly skips it in favour of letting data do the work.
- 02 Activation Functions Why depth is meaningless without a nonlinearity, and how the field moved from saturating sigmoids through ReLU and GELU to the gated SwiGLU that sits in almost every modern transformer feed-forward block.
- 03 Backpropagation and Automatic Differentiation How reverse-mode autodiff turns the chain rule into an efficient gradient algorithm, and the design choices PyTorch and JAX make to implement it at scale.
- 04 Convolutional Neural Networks Why weight sharing and local receptive fields make CNNs the right inductive bias for images, and where ViTs took over.
- 05 Generative Adversarial Networks The minimax game that dominated image generation for six years, why its training instability and mode collapse are structural rather than incidental, and what it left behind after diffusion overtook it.
- 06 Graph Neural Networks and Message Passing How one aggregate-and-update primitive covers most of the GNN literature, what the Weisfeiler-Lehman test says about its expressive ceiling, and why oversmoothing and oversquashing put a hard cap on depth.
- 07 Normalisation: BatchNorm, LayerNorm, RMSNorm Why normalisation accelerates training, why transformers use LayerNorm instead of BatchNorm, and why RMSNorm is now the default in Llama-class models.
- 08 Optimisers: SGD, Adam, AdamW, Lion How the standard optimiser stack evolved from plain SGD through Adam to memory-cheaper variants like Lion and Muon, and which learning-rate schedules actually work at scale.
- 09 Recurrent Networks: RNN, LSTM, GRU How gating fixed the vanishing-gradient problem in RNNs, and why transformers displaced them everywhere except streaming and on-device workloads.
- 10 Residual Connections and Skip Paths Why adding the input back to a layer's output is what makes networks of dozens or hundreds of layers trainable, and how the same trick underpins every modern transformer.
- 11 Self-Supervised Contrastive Learning How InfoNCE turns "these two crops came from the same photo" into a training signal strong enough to match supervised pretraining, and why every method in this family is fundamentally an anti-collapse mechanism.
- 12 State Space Models and Selective SSMs How a linear recurrence with a structured state matrix reaches transformer-level language modelling at linear cost and constant-memory decoding, what selectivity added, and the copying tasks that still expose the gap.
- 13 Variational Autoencoders and the ELBO How maximising an intractable data likelihood turns into maximising a tractable lower bound, why the reparameterisation trick is what makes that bound differentiable, and what posterior collapse costs you.