Concept library
1134 concepts across 20 domains and 101 tracks. Each track is a coherent sequence — read it top to bottom or dip in wherever the gap is.
All domains
01Foundations
02Transformer Internals
03Training & Fine-Tuning
04Reinforcement Learning
05Inference, Systems & Hardware
06Applied LLM Engineering
07Reasoning, Evaluation & Safety
08Multimodal & Applications
09Classical ML & Statistical Learning
10Causal Inference & Experimentation
11Time Series & Forecasting
12Graphs, Recommenders & Structured Data
13Generative Modelling Beyond Transformers
14Efficiency, Compression & Edge AI
15Search & Information Retrieval
16Data & Feature Engineering
17MLOps & Platform Engineering
18Security, Privacy & Adversarial ML
19Governance, Risk & Responsible AI
20Human-AI Interaction, Product & Economics
14
Efficiency, Compression & Edge AI
Making a model smaller, cheaper and local without giving away the thing that made it useful.
5tracks
51concepts
476cards
6.2hreading
Quantisation Post-training and quantisation-aware methods, outlier channels, GPTQ and AWQ, and low-bit arithmetic formats. 11 concepts · 104 cards
- 01 Calibration Data for Post-Training Quantisation The few hundred kilobytes of text that decide how a 70B model rounds, why the default recipe of 128 random web sequences is a convention rather than a result, and what the evidence says about choosing better.
- 02 Quantisation Grids, Scale and Zero Point The affine map between floating point and integers, why granularity is the single most consequential choice, and how clipping and rounding errors trade against each other.
- 03 When Quantisation Does Not Make Inference Faster Weight-only quantisation buys bandwidth, not arithmetic, so its speedup is a function of batch size and phase; the arithmetic that predicts where the gain disappears, and why a kernel can be slower than fp16.
- 04 AWQ and Activation-Aware Scaling The observation that weight importance is determined by activation magnitude rather than weight magnitude, and how a per-channel rescaling protects the important weights without keeping any of them in higher precision.
- 05 GPTQ and Second-Order Weight Rounding Why rounding each weight to its nearest grid point is not the best rounding, how the Hessian of the layer reconstruction error tells you what to do instead, and what the approximations cost.
- 06 KV Cache Quantisation Why the KV cache quantises along different axes for keys and values, how that asymmetry follows from where the outliers live, and why the cache is the one tensor whose quantisation error compounds over a generation.
- 07 Low-Bit Number Formats and Microscaling Why floating-point formats at 8 bits and below split differently between exponent and mantissa, what a shared block exponent buys, and how hardware support decides which format wins regardless of its numerical merits.
- 08 Mixed-Precision Bit Allocation and Sensitivity Why a fixed bit budget is better spent unevenly across layers, how Hessian-based sensitivity from HAWQ turns an exponential search into an integer programme, and where the second-order proxy and the hardware disagree.
- 09 Quantisation and Reasoning Models Reasoning models spend thousands of tokens per answer, which is exactly the regime where small per-token errors have room to compound; what the measurements show, where they disagree, and what to evaluate.
- 10 Quantisation-Aware Training and the Straight-Through Estimator How you backpropagate through a step function that has zero gradient everywhere, what QAT buys over post-training methods, and why it is used far less than its accuracy would justify.
- 11 Quantising Mixture-of-Experts Models Why MoE models are the most attractive quantisation target and the most fragile one, how the router turns rounding error into a discrete change of computation path, and what belongs in high precision.
Knowledge Distillation Soft targets and temperature, sequence-level and on-policy distillation, and when a student beats its teacher. 11 concepts · 101 cards
- 01 Feature and Attention Transfer in Distillation How FitNets hints, attention transfer, TinyBERT and MiniLM supervise a student's intermediate representations rather than only its outputs, what alignment problem each design has to solve, and the evidence that feature matching can make a student worse.
- 02 Multi-Teacher and Ensemble Distillation Why compressing an ensemble was distillation's original purpose, how averaging teachers in probability space differs from averaging logits, and why teacher disagreement is the only part of the signal that is worth paying for.
- 03 Temperature and Dark Knowledge What information a teacher's full output distribution carries that a hard label does not, why temperature is needed to expose it, and the gradient-scaling correction that everyone forgets.
- 04 Transfer Sets and Data-Free Distillation Distillation needs inputs to query the teacher on but not labels, which makes the choice of transfer set the dominant design decision, and makes distillation possible even when the original training data is gone.
- 05 Cross-Tokeniser Distillation Why a KL divergence between a teacher and a student with different tokenisers is not even well defined, the two distinct mismatches involved, and how optimal-transport and alignment-based losses get around them.
- 06 Distilling Rerankers into Retrievers How a cross-encoder's ranking signal is moved into a bi-encoder's parameters so it is paid at index time, why the target is a margin rather than a score, and why hard negatives decide whether any of it transfers.
- 07 Draft-Model Distillation for Speculative Decoding Why a draft model's standalone quality is the wrong objective, how distillation targets the acceptance rate directly, and why the training data has to come from the draft rather than the target.
- 08 On-Policy Distillation and Exposure Bias Why a student trained only on teacher trajectories cannot recover from its own errors, how generating from the student fixes the state distribution, and what reverse KL buys and costs.
- 09 Self-Distillation and Born-Again Networks The result that a student identical in architecture to its teacher often outperforms it, the competing explanations for why, and where the effect is genuinely useful rather than merely surprising.
- 10 The Capacity Gap in Distillation Why a stronger teacher can produce a worse student, what the intermediate-teacher fix does, and how to reason about the ratio between teacher and student capacity.
- 11 Token-Level Versus Sequence-Level Distillation Why matching a teacher's per-token distributions is not the same as matching its outputs, and how training on teacher-generated sequences changes the objective from mode-covering to mode-seeking.
Sparsity & Pruning Magnitude and second-order criteria, structured versus unstructured sparsity, and the hardware that rewards it. 11 concepts · 100 cards
- 01 Activation Sparsity in Feedforward Layers Trained transformers fire only a few percent of their MLP neurons on any given token, which is a property of the activations rather than the weights, and it is the reason a dense model can be computed sparsely without deleting anything.
- 02 Magnitude Pruning and the Lottery Ticket Hypothesis Why the simplest possible pruning criterion is so hard to beat, what the lottery ticket experiment actually claims, and the rewinding detail that decides whether it reproduces.
- 03 Structured Versus Unstructured Sparsity Why 90 percent unstructured sparsity can be slower than a dense matmul, what removing a whole channel buys instead, and how to decide which side of the tradeoff a deployment sits on.
- 04 Cashing In Activation Sparsity on Real Hardware Fifty percent activation sparsity converts to roughly fifteen percent latency and a 2.5x FLOP cut converts to 1.40x on a GPU, because decode is bandwidth-bound, gathers are not free, and batching makes the union of active neurons dense.
- 05 Contextual Sparsity and Activation Predictors If you knew in advance which neurons a token will activate you could avoid loading the rest, so the whole technique reduces to building a predictor that is cheaper than the work it saves and almost never wrong.
- 06 Depth Pruning and Layer Removal for LLMs Deleting whole transformer blocks is the one form of structured pruning that speeds up batch-one decoding almost linearly, and the evidence says deep layers are surprisingly removable for knowledge recall and surprisingly necessary for reasoning.
- 07 Dynamic Sparse Training Training a sparse network from scratch by continuously rewiring which weights exist, why the gradients of absent weights are the key signal, and what stops this from replacing dense training.
- 08 N:M Semi-Structured Sparsity The compromise pattern that hardware can accelerate, why 2:4 specifically, and the gap between the theoretical 2x and what a full model actually achieves.
- 09 Relufication and Sparsity-Aware Pretraining Swapping SiLU back to ReLU and continuing to pretrain recovers 90 percent activation sparsity that SwiGLU models never had, and the explicit top-K and dReLU variants show the quality cost of that sparsity is close to zero if you train for it.
- 10 Second-Order and Activation-Aware Pruning How the Hessian of the loss gives a principled importance score, why the exact version is intractable, and the two approximations that made one-shot pruning of large language models work.
- 11 Training-Free Activation Sparsification A magnitude threshold calibrated per layer gets 40 to 50 percent model-wide activation sparsity out of an off-the-shelf SwiGLU model with no retraining and no predictor network, which is a smaller win than relufication and costs nothing to try.
Efficient Architectures Small language models, depth-width tradeoffs, weight sharing, and architectures designed for a latency budget. 11 concepts · 97 cards
- 01 Depthwise Separable Convolutions and Mobile Networks Splitting a convolution into a per-channel spatial filter and a 1x1 channel mixer cuts its arithmetic by nearly an order of magnitude, which built MobileNet and EfficientNet, and also produced the field's clearest case of FLOPs failing to predict speed.
- 02 Per-Layer Embeddings and Parameter Offloading Why a model's parameter count stopped predicting its accelerator memory footprint, how per-layer embeddings move a large block of weights to cheap memory that only needs a lookup, and what the PCIe or unified-memory path costs when the trick is applied to the wrong tensors.
- 03 Small Language Models and the Overtraining Regime Why compute-optimal training is the wrong objective when inference dominates the bill, how far past Chinchilla the good small models actually go, and what stops the trend.
- 04 The Hardware Lottery in Architecture Choice Why architectural ideas succeed or fail partly on their fit to whatever silicon and kernels happen to exist, with the concrete cases (unstructured sparsity, state space models, capsule networks) where the fit rather than the idea decided the outcome.
- 05 Weight Tying and Parameter Sharing Where reusing one set of weights in several places is nearly free, where it costs real capability, and why the embedding matrix is the case everyone gets right and the layer stack is the case everyone gets wrong.
- 06 Architecting to a Latency Budget How to design a model backwards from a millisecond target using arithmetic intensity, why parameter count is the wrong currency, and the design moves that actually reduce time to first and subsequent tokens.
- 07 Cross-Layer KV Sharing Why sharing key and value tensors between layers is the one KV cache reduction that composes with GQA and sliding windows, how CLA and YOCO differ in what they let you skip, and the prefill and quality costs that come with each layout.
- 08 Depth Versus Width Tradeoffs Why two models with identical parameter counts behave differently depending on how the parameters are arranged, what depth buys that width cannot, and how the hardware votes for width.
- 09 Hardware-Aware Architecture Search Why optimising a model for FLOPs produces slow models, how measured latency became the objective instead, and what makes once-for-all supernet training the practical form of the idea.
- 10 Looped Transformers and Recurrent Depth What you get when the same transformer block runs several times instead of stacking distinct layers, why depth rather than parameter count is what many reasoning problems need, and the memory-bandwidth and interpretability costs of buying compute this way.
- 11 Nested and Elastic Transformers How training one model with deliberately nested sub-networks lets you extract a whole family of deployable models at no extra training cost, what Mix'n'Match buys over a fixed ladder of sizes, and why the parent still has to be resident in memory for the child to run.
On-Device & Edge AI Mobile NPUs, memory-bound inference on consumer silicon, compilation targets and privacy-driven local models. 7 concepts · 74 cards
- 01 Cloud-Edge Split Inference Where to cut a model between device and datacenter is an optimisation over per-layer compute, activation size and link conditions; it works well for feed-forward vision models and poorly for autoregressive LLMs, which split at the query instead.
- 02 Compilation Targets and Runtime Fragmentation The path from a trained PyTorch model to something that runs on a phone, why the intermediate format is where most deployment failures happen, and what each of the major runtimes assumes.
- 03 Mobile NPUs and the Accelerator Zoo What a phone's neural processing unit is good at, why the same model runs at wildly different speeds on the CPU, GPU and NPU of one device, and the fallback that silently destroys performance.
- 04 Privacy-Driven Local Inference What running a model locally actually guarantees, which parts of the pipeline still leak, and how hybrid designs preserve most of the property while escalating the hard requests.
- 05 Federated Learning on Edge Devices Training a shared model without collecting the data, why non-IID client distributions break the averaging assumption, and the systems constraints that decide what is actually trainable.
- 06 On-Device LLM Inference Constraints The arithmetic that decides whether a language model can run on a phone, why bandwidth rather than compute is the binding constraint, and what the KV cache does to the memory budget.
- 07 TinyML and Microcontroller Inference On a microcontroller the binding constraint is not compute but two memories, flash for weights and code and SRAM for activations, and the techniques that work (static arenas, peak-memory-aware search, patch-based execution) all attack peak SRAM.