PII Detection and Removal
Scrubbing personally identifiable information from web-scale corpora before LLM pretraining reduces memorisation risk and legal exposure, but every detection method trades recall against corpus damage.
A structured path from linear algebra to production agents. Every concept is written to be read once and understood, then drilled with flashcards until it sticks, then extended with a deep dive that shows how it plays out in real systems.
vLLM's September release lets a client ask, per request, what fraction of its tokens a small draft model guessed right. The same stack cuts the draft length to zero above 128 concurrent requests. Both are the same admission.
Speculative decoding never made a model faster; it converts a server's idle arithmetic into lower latency, and because batching spends that same arithmetic, the stack now rations the speedup to the most predictable requests first and to nobody once the machine is full.
A concept, a flashcard, a quiz and a deep dive, picked from the date itself so everybody sees the same four and a reload does not reshuffle them. Each one can be shared whole, with the link, to X, LinkedIn, WhatsApp or Substack.
Scrubbing personally identifiable information from web-scale corpora before LLM pretraining reduces memorisation risk and legal exposure, but every detection method trades recall against corpus damage.
Why is rolling-origin evaluation the honest procedure, and why is it skipped?
Click to flipBecause a single train-test split gives one noisy estimate and random splits leak the future, while rolling origin refits and forecasts forward repeatedly to give a distribution of errors over time. It is skipped because it costs many refits, which for a large global model is a substantial compute bill.
A. N-Gram Models and Smoothing
The pre-neural language model that ruled for three decades, why counting words breaks the moment you hit a sequence you have never seen, and the smoothing tricks invented to patch that hole.
B. The Softmax Bottleneck
Why a standard softmax output layer is a low-rank approximation to the true distribution of language, the "bottleneck" that caps what any single softmax can express, and the mixture trick that breaks it.
C. f-Divergences Beyond KL
the answer
KL is one member of a family generated by a convex function, the choice of member decides whether your model covers the data or collapses onto a mode, and some tasks need a divergence that is not in the family at all.
D. Channel Capacity and the Noisy Channel
Shannon's capacity theorem, why it says reliable communication is possible at any rate below capacity and impossible above it, and why the noisy-channel decomposition keeps reappearing in language modelling.
Training a model to generate photorealistic images teaches it geometry, semantics, depth, and object relationships. Google DeepMind's Vision Banana proves that a lightweight instruction-tuning pass over an image generator can beat SAM 3 on segmentation and De…
1015 self-contained explainers organised into 20 domains and 101 tracks. Each one is a short read that starts from what you already know and ends somewhere useful.
Open the tree → 0211,105 cards drawn straight from the concepts. Flip with a keypress, shuffle a deck, mark what you know. Progress is kept in your browser, so nothing to sign up for.
Start a deck → 03123 long-form pieces with the maths, the diagrams and the numbers intact. These are the articles to read once you know the vocabulary and want to know why the field settled where it did.
Read the writing →The order is deliberate. Domain 01 assumes nothing; domain 08 assumes the rest. You do not have to follow it, but if you are starting cold, start at the top.
The mathematics and neural-network mechanics everything else assumes.
Open the box. How a language model actually turns text into predictions.
From raw web crawl to an aligned model — data, dynamics, scale and adaptation.
Classical RL, then the specific dialect of it that post-trains language models.
Where the model meets the silicon, the memory bus and the latency budget.
Building things people use: retrieval, prompts, agents and production architecture.
Models that think longer, the evals that measure them, and the failure modes that matter.
Beyond text — vision, speech, robotics and scientific discovery.
The statistics and non-neural models that still decide most production predictions.
Telling correlation from cause, and measuring whether a change actually helped.
Data with an arrow of time, where shuffling the rows destroys the problem.
Learning over relations, catalogues and columns rather than free text.
Diffusion, flows, adversarial games and state space models, and the theory that connects them.
Making a model smaller, cheaper and local without giving away the thing that made it useful.
Thirty years of ranking research that RAG rediscovered, usually the hard way.
The pipelines, formats and contracts that decide whether a model ever sees correct inputs.
Everything between a notebook that works and a system that keeps working.
Attacks on models, data and the supply chain, and the defences that survive contact.
Frameworks, regulation and audit evidence, treated as engineering rather than paperwork.
The people using the system, the product decisions around it, and what compute actually costs.
Beginner-level concepts that assume no prior machine learning.
The exact sequence of operations inside one transformer block, from tensor shapes to parameter counts, and why every frontier model is just this same function stacked dozens of times.
Attention InternalsThe mental model that makes attention click before the matrix algebra does - a lookup table where the key match is a matter of degree, not an exact hit or miss.
Speech RecognitionRaw audio waveforms are rarely fed directly to speech models; this concept explains how and why they are first converted into spectrogram-based representations that compress perceptual information into a learnable 2-D grid.
Context & In-Context LearningHow a language model turns next-token prediction into a paragraph, and why generation is a loop that feeds its own output back as input.
Tensors & Neural PlumbingBroadcasting is the rule that lets a bias vector add to every row of a batch without an explicit loop, and understanding its shape-matching logic prevents the class of bugs that produce wrong answers without an error.
Prompt EngineeringWhy telling the model to think step by step radically improves reasoning, and when it actively hurts.
Statistical InferenceWhat the 95% in a 95% interval refers to, why the Wald interval for a proportion is badly behaved near zero, and how a confidence interval differs from the credible interval people usually think they are reading.
Embeddings & RepresentationsTwo near-identical looking formulas that answer different questions, one measures direction alone, the other measures direction and magnitude together, and picking the wrong one silently breaks a search or ranking system.
An H100 advertises 989 teraflops. Generating one token from an 8B model uses roughly 0.3% of that. The gap is not a bug in your code or a missing compiler flag; it is a single ratio, FLOPs …
Training & AlignmentThe industry's defence against data poisoning was arithmetic: an attacker needs a percentage of the corpus, and a percentage of 260 billion tokens is unobtainable. In October 2025 the large…
Safety, Security & GovernanceIn 2014 a perturbation the size of one 8-bit colour step turned a 57.7 percent panda into a 99.3 percent gibbon. Twelve years, 300 million synthetic training images and more than 10^21 trai…
Platforms & PracticeModel registries were built to version a trained artefact you own. An LLM application's behaviour comes from a hosted snapshot that retires on someone else's calendar, plus prompts, an inde…
Model ArchitectureA million-token window promises perfect recall of everything you feed it. Controlled tests on 18 frontier models show recall degrading steadily, unevenly, and well before the window fills, …
Reasoning & EvaluationIn 2002 Jon Kleinberg proved that no clustering function can satisfy three properties almost everyone would ask for. Every algorithm is therefore a definition of what a cluster is, and ever…