Concept library
1015 concepts across 20 domains and 101 tracks. Each track is a coherent sequence — read it top to bottom or dip in wherever the gap is.
All domains
01Foundations
02Transformer Internals
03Training & Fine-Tuning
04Reinforcement Learning
05Inference, Systems & Hardware
06Applied LLM Engineering
07Reasoning, Evaluation & Safety
08Multimodal & Applications
09Classical ML & Statistical Learning
10Causal Inference & Experimentation
11Time Series & Forecasting
12Graphs, Recommenders & Structured Data
13Generative Modelling Beyond Transformers
14Efficiency, Compression & Edge AI
15Search & Information Retrieval
16Data & Feature Engineering
17MLOps & Platform Engineering
18Security, Privacy & Adversarial ML
19Governance, Risk & Responsible AI
20Human-AI Interaction, Product & Economics
01
Foundations
The mathematics and neural-network mechanics everything else assumes.
4tracks
63concepts
707cards
8.4hreading
Mathematical Foundations Linear algebra, probability, calculus and optimisation — the machinery every model is built on. 16 concepts · 223 cards
- 01 Calculus and Gradients Partial derivatives, the chain rule as the engine of backprop, why second-order methods are rare in deep learning, and what gradient clipping actually does.
- 02 Fourier Analysis for Neural Models How the convolution theorem turns quadratic sequence mixing into n log n, why MLPs refuse to learn high frequencies, and why rotary position encoding is a frequency argument wearing a rotation costume.
- 03 Linear Algebra for ML Vector spaces, matrix decompositions, and why low-rank structure underlies LoRA, PCA, and the quantisation tricks that make modern LLMs cheap to serve.
- 04 Numerical Computation Gotchas Catastrophic cancellation, the log-sum-exp trick, mixed-precision training, the determinism tax, and how to actually debug a NaN in a 70B model.
- 05 Probability and Information Theory Distributions, expectations, entropy, KL, and why softmax + cross-entropy is the canonical pair that secretly underlies almost every LLM loss.
- 06 Concentration Inequalities and Generalisation Bounds The tools that turn "the average of many random things is close to its mean" into explicit numbers, how they build classical generalisation bounds, and why those bounds are vacuous for deep networks.
- 07 Fisher Information and Natural Gradient Why the steepest descent direction depends on the coordinate system you happened to choose, how the Fisher matrix fixes that, and which popular optimisers are approximating it badly.
- 08 Group Equivariance and Symmetry When a task has a symmetry, building it into the architecture instead of learning it from data cuts sample complexity, and the group-theoretic formulation says exactly how to do that.
- 09 Lagrangian Duality and Constrained Optimisation How a constraint becomes a penalty with a price attached, why the KL-regularised objective at the heart of RLHF has a closed-form solution, and where duality quietly fails.
- 10 Loss Landscape Geometry and Mode Connectivity Independently trained networks are not isolated in separate valleys; they are connected by low-loss paths, and after undoing permutation symmetry they are often in the same basin.
- 11 Optimal Transport and Wasserstein Distances Why KL divergence is useless between distributions that do not overlap, how the cost of moving mass gives a metric that is not, and what entropic regularisation trades away to make it computable.
- 12 Optimisation Theory Convexity, why SGD finds good solutions on non-convex losses, saddle points at scale, momentum as a damped oscillator, and learning-rate schedules as implicit regularisation.
- 13 SGD as a Stochastic Differential Equation Treating mini-batch gradient descent as a noisy continuous-time process explains the linear scaling rule, why the learning-rate-to-batch-size ratio is the real hyperparameter, and where the analogy fails.
- 14 Spectral Analysis of Weight Matrices What the singular values of a weight matrix control, why random matrix theory predicts the spectrum at initialisation, and how spectral thinking connects initialisation, LoRA and modern optimisers.
- 15 Statistical Learning Theory Primer Bias-variance, PAC-learning, VC dimension, why deep nets break classical generalisation bounds, double descent, and what scaling laws are actually saying.
- 16 The Manifold Hypothesis and Intrinsic Dimension Why high-dimensional data concentrates on a low-dimensional surface, how to estimate that surface's dimension from samples, and why the number predicts how much data a model will need.
Tensors & Neural Plumbing Shapes, matmuls, forward and backward passes, parameter counts, memory footprints. 18 concepts · 211 cards
- 01 Broadcasting and Vectorisation Broadcasting is the rule that lets a bias vector add to every row of a batch without an explicit loop, and understanding its shape-matching logic prevents the class of bugs that produce wrong answers without an error.
- 02 Matrix Multiplication: The Core Operation Nearly every FLOP a transformer spends is a matrix multiplication; understanding its shape rule and its cost is the single most load-bearing piece of maths in deep learning.
- 03 Tensors, Shapes, and Batching Every number a transformer touches lives inside a tensor with a fixed shape; learning to read and predict those shapes is the fastest way to stop being confused by model code.
- 04 Why Non-Linearity Matters Stack any number of linear layers and the result is still one linear layer; the humble activation function is the only thing standing between a transformer and a glorified matrix multiplication.
- 05 Checkpoint Formats and State Dicts A checkpoint is a dictionary from parameter names to tensors plus a pile of implicit assumptions about dtype, sharding, and key naming, and every one of those assumptions is somewhere a load goes wrong.
- 06 Counting Transformer Parameters A transformer's parameter count is not a mysterious headline number, it is the sum of a handful of matrix shapes multiplied out, and knowing the formula lets you sanity-check any model card in seconds.
- 07 Einsum and Tensor Contractions Einstein summation notation expresses matmuls, batched matmuls, and attention itself as one uniform pattern, and reading it fluently is the fastest way to understand what a line of unfamiliar model code actually computes.
- 08 Forward-Mode and Reverse-Mode Autodiff Backpropagation is one of two ways to apply the chain rule mechanically, and the choice between them is decided by a single number, the ratio of inputs to outputs, which is why training uses reverse mode and Hessian-vector products use both.
- 09 Padding, Masks, and Variable-Length Batching Real batches contain sequences of different lengths, and the three ways of handling that difference, padding, packing, and varlen kernels, have wildly different costs and each has a signature bug.
- 10 Tensor Memory Layout and Contiguity A tensor is a pointer, a shape, and a stride tuple; transposes and slices change only the metadata, which is why some reshapes are free, some silently copy gigabytes, and a permute in the wrong place can halve your throughput.
- 11 The Backward Pass and Gradient Flow Backpropagation through a transformer is the forward pass run in reverse with the chain rule attached; seeing which operations preserve gradient magnitude and which shrink it explains why architecture choices exist.
- 12 The Forward Pass End to End Trace one token's numbers from embedding table to output logits and every "mysterious" transformer component turns out to be a shape-preserving or shape-mixing step in a fixed pipeline.
- 13 Views, Aliasing, and In-Place Operations A transpose returns a new tensor object that shares its old storage, which is why in-place edits leak across variables you thought were separate and why autograd raises "a variable needed for gradient computation has been modified".
- 14 Accumulation Precision and Mixed-Dtype Plumbing Storing weights in 16 bits is safe because almost nothing is actually computed in 16 bits — matmuls accumulate in fp32, reductions run in fp32, and the master copy is fp32, and every documented failure of low-precision training is one of these rules being broken.
- 15 Batch Invariance and Numerical Reproducibility Temperature zero does not give the same answer twice, and the reason is neither random seeds nor GPU atomics; it is that reduction kernels change their split strategy with batch shape, so your logits depend on which other requests happened to be in flight.
- 16 Signal Propagation and Initialisation How a network's weights are initialised decides, before a single gradient step, whether activations and gradients stay in a trainable range or collapse to zero or explode across depth.
- 17 Training Memory Footprint A model's weights are the smallest part of its training memory bill; optimiser state, gradients, and activations usually cost several times more, and knowing the breakdown explains why training needs far more memory than inference.
- 18 muP and Hyperparameter Transfer Under standard parametrisation the best learning rate drifts as a model gets wider, so every scale-up is a fresh search; muP rescales initialisation, learning rates, and multipliers per layer so the optimum stays put and can be tuned on a small proxy model.
Deep Learning Building Blocks Convolutions, recurrence, normalisation, activations, optimisers and regularisation. 13 concepts · 112 cards
- 01 Dropout and Modern Regularisation Why dropout was the dominant regulariser for a decade and why modern LLM training mostly skips it in favour of letting data do the work.
- 02 Activation Functions Why depth is meaningless without a nonlinearity, and how the field moved from saturating sigmoids through ReLU and GELU to the gated SwiGLU that sits in almost every modern transformer feed-forward block.
- 03 Backpropagation and Automatic Differentiation How reverse-mode autodiff turns the chain rule into an efficient gradient algorithm, and the design choices PyTorch and JAX make to implement it at scale.
- 04 Convolutional Neural Networks Why weight sharing and local receptive fields make CNNs the right inductive bias for images, and where ViTs took over.
- 05 Generative Adversarial Networks The minimax game that dominated image generation for six years, why its training instability and mode collapse are structural rather than incidental, and what it left behind after diffusion overtook it.
- 06 Graph Neural Networks and Message Passing How one aggregate-and-update primitive covers most of the GNN literature, what the Weisfeiler-Lehman test says about its expressive ceiling, and why oversmoothing and oversquashing put a hard cap on depth.
- 07 Normalisation: BatchNorm, LayerNorm, RMSNorm Why normalisation accelerates training, why transformers use LayerNorm instead of BatchNorm, and why RMSNorm is now the default in Llama-class models.
- 08 Optimisers: SGD, Adam, AdamW, Lion How the standard optimiser stack evolved from plain SGD through Adam to memory-cheaper variants like Lion and Muon, and which learning-rate schedules actually work at scale.
- 09 Recurrent Networks: RNN, LSTM, GRU How gating fixed the vanishing-gradient problem in RNNs, and why transformers displaced them everywhere except streaming and on-device workloads.
- 10 Residual Connections and Skip Paths Why adding the input back to a layer's output is what makes networks of dozens or hundreds of layers trainable, and how the same trick underpins every modern transformer.
- 11 Self-Supervised Contrastive Learning How InfoNCE turns "these two crops came from the same photo" into a training signal strong enough to match supervised pretraining, and why every method in this family is fundamentally an anti-collapse mechanism.
- 12 State Space Models and Selective SSMs How a linear recurrence with a structured state matrix reaches transformer-level language modelling at linear cost and constant-memory decoding, what selectivity added, and the copying tasks that still expose the gap.
- 13 Variational Autoencoders and the ELBO How maximising an intractable data likelihood turns into maximising a tractable lower bound, why the reparameterisation trick is what makes that bound differentiable, and what posterior collapse costs you.
Information Theory for Language Entropy, cross-entropy, KL, perplexity, calibration, and language modelling as compression. 16 concepts · 161 cards
- 01 Log-Probs and Token Probabilities What the log-probability numbers an API actually returns mean, how to turn them back into probabilities and perplexity, and the practical uses (reranking, confidence checks, hallucination flags) that make them worth reading instead of skipping.
- 02 N-Gram Models and Smoothing The pre-neural language model that ruled for three decades, why counting words breaks the moment you hit a sequence you have never seen, and the smoothing tricks invented to patch that hole.
- 03 Perplexity - Measuring a Language Model The oldest language-model metric, what "perplexity 12" actually means, and why you can never compare it across two models with different tokenisers.
- 04 Bits-per-Byte The tokenizer-invariant way to report a language model's compression performance, how to compute it from cross-entropy, and why serious scaling-law and benchmark papers report it instead of perplexity.
- 05 Cross-Entropy and KL Divergence The precise relationship between cross-entropy, entropy, and KL divergence, why minimising one is minimising "excess bits," and why forward and reverse KL pull a model in opposite directions.
- 06 Entropy and Surprise Why "surprising" has an exact numerical meaning, how -log p(x) turns probability into a unit of information, and what a language model's entropy floor tells you about a piece of text.
- 07 Maximum-Likelihood Estimation The estimation principle underneath every LLM's training objective, why "maximise probability of the data" and "minimise cross-entropy" are the exact same optimisation, and what changes when you add a prior.
- 08 Calibration of Language Models What it means for a model's stated confidence to be trustworthy, why pretrained next-token probabilities start out well calibrated and RLHF quietly breaks that, and how to actually measure the gap.
- 09 Channel Capacity and the Noisy Channel Shannon's capacity theorem, why it says reliable communication is possible at any rate below capacity and impossible above it, and why the noisy-channel decomposition keeps reappearing in language modelling.
- 10 Kolmogorov Complexity and MDL The shortest program that outputs a string is the ultimate measure of its information content, why it is uncomputable, and how minimum description length turns that uncomputable ideal into a usable model selection principle.
- 11 Language Modelling as Compression Every autoregressive language model is, exactly and not metaphorically, a lossless compressor; arithmetic coding is the bridge, and that equivalence is what let a text-only LLM out-compress PNG and FLAC on images and audio.
- 12 Mutual Information and Representations The information-theoretic quantity that measures how much one variable tells you about another, why it is the right lens for judging whether a learned representation actually captured something, and why estimating it in high dimensions is notoriously hard.
- 13 Rate-Distortion Theory The theory of how few bits a source needs when perfect reconstruction is not required, why every compression decision in an LLM stack is a point on one curve, and what the curve's shape tells you.
- 14 The Softmax Bottleneck Why a standard softmax output layer is a low-rank approximation to the true distribution of language, the "bottleneck" that caps what any single softmax can express, and the mixture trick that breaks it.
- 15 Typical Sets and the AEP Why almost all probability mass sits on a vanishingly small set of sequences that are individually unremarkable, and why the most likely sequence is usually not a typical one.
- 16 f-Divergences Beyond KL KL is one member of a family generated by a convex function, the choice of member decides whether your model covers the data or collapses onto a mode, and some tasks need a divergence that is not in the family at all.