Concept library
446 concepts across 8 domains and 36 tracks. Each track is a coherent sequence — read it top to bottom or dip in wherever the gap is.
01
Foundations
The mathematics and neural-network mechanics everything else assumes.
4tracks
35concepts
170cards
4.6hreading
Mathematical Foundations Linear algebra, probability, calculus and optimisation — the machinery every model is built on. 6 concepts · 30 cards
- 01 Calculus and Gradients Partial derivatives, the chain rule as the engine of backprop, why second-order methods are rare in deep learning, and what gradient clipping actually does.
- 02 Linear Algebra for ML Vector spaces, matrix decompositions, and why low-rank structure underlies LoRA, PCA, and the quantisation tricks that make modern LLMs cheap to serve.
- 03 Numerical Computation Gotchas Catastrophic cancellation, the log-sum-exp trick, mixed-precision training, the determinism tax, and how to actually debug a NaN in a 70B model.
- 04 Probability and Information Theory Distributions, expectations, entropy, KL, and why softmax + cross-entropy is the canonical pair that secretly underlies almost every LLM loss.
- 05 Optimisation Theory Convexity, why SGD finds good solutions on non-convex losses, saddle points at scale, momentum as a damped oscillator, and learning-rate schedules as implicit regularisation.
- 06 Statistical Learning Theory Primer Bias-variance, PAC-learning, VC dimension, why deep nets break classical generalisation bounds, double descent, and what scaling laws are actually saying.
Tensors & Neural Plumbing Shapes, matmuls, forward and backward passes, parameter counts, memory footprints. 10 concepts · 42 cards
- 01 Broadcasting and Vectorisation Broadcasting is the rule that lets a bias vector add to every row of a batch without an explicit loop, and understanding its shape-matching logic prevents the class of bugs that produce wrong answers without an error.
- 02 Matrix Multiplication: The Core Operation Nearly every FLOP a transformer spends is a matrix multiplication; understanding its shape rule and its cost is the single most load-bearing piece of maths in deep learning.
- 03 Tensors, Shapes, and Batching Every number a transformer touches lives inside a tensor with a fixed shape; learning to read and predict those shapes is the fastest way to stop being confused by model code.
- 04 Why Non-Linearity Matters Stack any number of linear layers and the result is still one linear layer; the humble activation function is the only thing standing between a transformer and a glorified matrix multiplication.
- 05 Counting Transformer Parameters A transformer's parameter count is not a mysterious headline number, it is the sum of a handful of matrix shapes multiplied out, and knowing the formula lets you sanity-check any model card in seconds.
- 06 Einsum and Tensor Contractions Einstein summation notation expresses matmuls, batched matmuls, and attention itself as one uniform pattern, and reading it fluently is the fastest way to understand what a line of unfamiliar model code actually computes.
- 07 The Backward Pass and Gradient Flow Backpropagation through a transformer is the forward pass run in reverse with the chain rule attached; seeing which operations preserve gradient magnitude and which shrink it explains why architecture choices exist.
- 08 The Forward Pass End to End Trace one token's numbers from embedding table to output logits and every "mysterious" transformer component turns out to be a shape-preserving or shape-mixing step in a fixed pipeline.
- 09 Signal Propagation and Initialisation How a network's weights are initialised decides, before a single gradient step, whether activations and gradients stay in a trainable range or collapse to zero or explode across depth.
- 10 Training Memory Footprint A model's weights are the smallest part of its training memory bill; optimiser state, gradients, and activations usually cost several times more, and knowing the breakdown explains why training needs far more memory than inference.
Deep Learning Building Blocks Convolutions, recurrence, normalisation, activations, optimisers and regularisation. 8 concepts · 37 cards
- 01 Dropout and Modern Regularisation Why dropout was the dominant regulariser for a decade and why modern LLM training mostly skips it in favour of letting data do the work.
- 02 Activation Functions Why depth is meaningless without a nonlinearity, and how the field moved from saturating sigmoids through ReLU and GELU to the gated SwiGLU that sits in almost every modern transformer feed-forward block.
- 03 Backpropagation and Automatic Differentiation How reverse-mode autodiff turns the chain rule into an efficient gradient algorithm, and the design choices PyTorch and JAX make to implement it at scale.
- 04 Convolutional Neural Networks Why weight sharing and local receptive fields make CNNs the right inductive bias for images, and where ViTs took over.
- 05 Normalisation: BatchNorm, LayerNorm, RMSNorm Why normalisation accelerates training, why transformers use LayerNorm instead of BatchNorm, and why RMSNorm is now the default in Llama-class models.
- 06 Optimisers: SGD, Adam, AdamW, Lion How the standard optimiser stack evolved from plain SGD through Adam to memory-cheaper variants like Lion and Muon, and which learning-rate schedules actually work at scale.
- 07 Recurrent Networks: RNN, LSTM, GRU How gating fixed the vanishing-gradient problem in RNNs, and why transformers displaced them everywhere except streaming and on-device workloads.
- 08 Residual Connections and Skip Paths Why adding the input back to a layer's output is what makes networks of dozens or hundreds of layers trainable, and how the same trick underpins every modern transformer.
Information Theory for Language Entropy, cross-entropy, KL, perplexity, calibration, and language modelling as compression. 11 concepts · 61 cards
- 01 Log-Probs and Token Probabilities What the log-probability numbers an API actually returns mean, how to turn them back into probabilities and perplexity, and the practical uses (reranking, confidence checks, hallucination flags) that make them worth reading instead of skipping.
- 02 N-Gram Models and Smoothing The pre-neural language model that ruled for three decades, why counting words breaks the moment you hit a sequence you have never seen, and the smoothing tricks invented to patch that hole.
- 03 Perplexity - Measuring a Language Model The oldest language-model metric, what "perplexity 12" actually means, and why you can never compare it across two models with different tokenisers.
- 04 Bits-per-Byte The tokenizer-invariant way to report a language model's compression performance, how to compute it from cross-entropy, and why serious scaling-law and benchmark papers report it instead of perplexity.
- 05 Cross-Entropy and KL Divergence The precise relationship between cross-entropy, entropy, and KL divergence, why minimising one is minimising "excess bits," and why forward and reverse KL pull a model in opposite directions.
- 06 Entropy and Surprise Why "surprising" has an exact numerical meaning, how -log p(x) turns probability into a unit of information, and what a language model's entropy floor tells you about a piece of text.
- 07 Maximum-Likelihood Estimation The estimation principle underneath every LLM's training objective, why "maximise probability of the data" and "minimise cross-entropy" are the exact same optimisation, and what changes when you add a prior.
- 08 Calibration of Language Models What it means for a model's stated confidence to be trustworthy, why pretrained next-token probabilities start out well calibrated and RLHF quietly breaks that, and how to actually measure the gap.
- 09 Language Modelling as Compression Every autoregressive language model is, exactly and not metaphorically, a lossless compressor; arithmetic coding is the bridge, and that equivalence is what let a text-only LLM out-compress PNG and FLAC on images and audio.
- 10 Mutual Information and Representations The information-theoretic quantity that measures how much one variable tells you about another, why it is the right lens for judging whether a learned representation actually captured something, and why estimating it in high dimensions is notoriously hard.
- 11 The Softmax Bottleneck Why a standard softmax output layer is a low-rank approximation to the true distribution of language, the "bottleneck" that caps what any single softmax can express, and the mixture trick that breaks it.