Mathematical Foundations
Linear algebra, probability, calculus and optimisation — the machinery every model is built on.
16concepts
223flashcards
140minutes of reading
- 01 Concentration Inequalities and Generalisation Bounds The tools that turn "the average of many random things is close to its mean" into explicit numbers, how they build classical generalisation bounds, and why those bounds are vacuous for deep networks.
- 02 Fisher Information and Natural Gradient Why the steepest descent direction depends on the coordinate system you happened to choose, how the Fisher matrix fixes that, and which popular optimisers are approximating it badly.
- 03 Group Equivariance and Symmetry When a task has a symmetry, building it into the architecture instead of learning it from data cuts sample complexity, and the group-theoretic formulation says exactly how to do that.
- 04 Lagrangian Duality and Constrained Optimisation How a constraint becomes a penalty with a price attached, why the KL-regularised objective at the heart of RLHF has a closed-form solution, and where duality quietly fails.
- 05 Loss Landscape Geometry and Mode Connectivity Independently trained networks are not isolated in separate valleys; they are connected by low-loss paths, and after undoing permutation symmetry they are often in the same basin.
- 06 Optimal Transport and Wasserstein Distances Why KL divergence is useless between distributions that do not overlap, how the cost of moving mass gives a metric that is not, and what entropic regularisation trades away to make it computable.
- 07 Optimisation Theory Convexity, why SGD finds good solutions on non-convex losses, saddle points at scale, momentum as a damped oscillator, and learning-rate schedules as implicit regularisation.
- 08 SGD as a Stochastic Differential Equation Treating mini-batch gradient descent as a noisy continuous-time process explains the linear scaling rule, why the learning-rate-to-batch-size ratio is the real hyperparameter, and where the analogy fails.
- 09 Spectral Analysis of Weight Matrices What the singular values of a weight matrix control, why random matrix theory predicts the spectrum at initialisation, and how spectral thinking connects initialisation, LoRA and modern optimisers.
- 10 Statistical Learning Theory Primer Bias-variance, PAC-learning, VC dimension, why deep nets break classical generalisation bounds, double descent, and what scaling laws are actually saying.
- 11 The Manifold Hypothesis and Intrinsic Dimension Why high-dimensional data concentrates on a low-dimensional surface, how to estimate that surface's dimension from samples, and why the number predicts how much data a model will need.