Mathematical Foundations
Linear algebra, probability, calculus and optimisation — the machinery every model is built on.
6concepts
30flashcards
54minutes of reading
- 01 Calculus and Gradients Partial derivatives, the chain rule as the engine of backprop, why second-order methods are rare in deep learning, and what gradient clipping actually does.
- 02 Linear Algebra for ML Vector spaces, matrix decompositions, and why low-rank structure underlies LoRA, PCA, and the quantisation tricks that make modern LLMs cheap to serve.
- 03 Numerical Computation Gotchas Catastrophic cancellation, the log-sum-exp trick, mixed-precision training, the determinism tax, and how to actually debug a NaN in a 70B model.
- 04 Probability and Information Theory Distributions, expectations, entropy, KL, and why softmax + cross-entropy is the canonical pair that secretly underlies almost every LLM loss.
- 05 Optimisation Theory Convexity, why SGD finds good solutions on non-convex losses, saddle points at scale, momentum as a damped oscillator, and learning-rate schedules as implicit regularisation.
- 06 Statistical Learning Theory Primer Bias-variance, PAC-learning, VC dimension, why deep nets break classical generalisation bounds, double descent, and what scaling laws are actually saying.