Concept library
446 concepts across 8 domains and 36 tracks. Each track is a coherent sequence — read it top to bottom or dip in wherever the gap is.
03
Training & Fine-Tuning
From raw web crawl to an aligned model — data, dynamics, scale and adaptation.
6tracks
102concepts
687cards
13.1hreading
Pretraining Data Pipelines Web-scale corpus construction, filtering, deduplication, decontamination and data mixtures. 20 concepts · 140 cards
- 01 Benchmark Decontamination Benchmark decontamination is the process of identifying and removing evaluation set examples from a model's pretraining corpus so that reported benchmark scores reflect genuine generalisation rather than memorised answers.
- 02 Curriculum and Data Ordering How the sequence and mixture proportions of training batches affect what an LLM learns and when, and why naive i.i.d. sampling often leaves capability on the table.
- 03 Data Provenance and Licensing Data provenance tracks the origin and transformation history of every byte in a training corpus, while licensing determines whether you are legally permitted to use it at all.
- 04 Deduplication and Memorisation Duplicate training examples cause language models to memorise and verbatim-reproduce training text, so removing duplicates before training reduces regurgitation, improves generalisation, and lowers compute waste.
- 05 Document-Level vs Token-Level Deduplication Document-level deduplication removes whole near-duplicate pages using hashing, while token-level deduplication removes repeated spans within and across documents using suffix arrays, each with distinct cost-quality trade-offs in web-scale corpus construction.
- 06 Exact and Fuzzy Deduplication How hash-based exact matching and MinHash locality-sensitive hashing together remove the duplicate content that otherwise inflates memorisation and wastes training compute.
- 07 Heuristic Quality Filters Heuristic quality filters are rule-based passes over raw web text that remove boilerplate, malformed documents, and low-information content before any classifier is trained or applied.
- 08 Language Identification and Filtering Language identification assigns a language label to each document in a web crawl so pipelines can retain target-language text and discard everything else before quality filtering begins.
- 09 MinHash and LSH for Deduplication MinHash estimates Jaccard similarity between documents in constant memory, and LSH bucketing turns that estimate into a sub-linear nearest-neighbour search, making corpus-scale near-deduplication tractable.
- 10 Multilingual Data Balancing Multilingual data balancing decides how many tokens from each language land in a pretraining corpus, trading raw web proportions against deliberate upsampling to serve low-resource languages without degrading high-resource ones.
- 11 PII Detection and Removal Scrubbing personally identifiable information from web-scale corpora before LLM pretraining reduces memorisation risk and legal exposure, but every detection method trades recall against corpus damage.
- 12 Quality Filtering with Classifiers Classifier-based quality filtering uses lightweight models trained on curated reference corpora to score and discard low-quality web documents before LLM pretraining.
- 13 Text Extraction and Boilerplate Removal Converting raw HTML from web crawls into clean, main-content text is a lossy signal-recovery problem, and the choices made here propagate irreversibly through every downstream filtering and training stage.
- 14 Toxicity Filtering of Pretraining Data Toxicity filtering removes hate speech, obscene language, and harmful content from web-crawled corpora before LLM pretraining, using blocklists, classifiers, and API-based scoring, each with distinct recall/precision trade-offs and equity side effects.
- 15 Training a Tokeniser on a Corpus Training a tokeniser on your pretraining corpus, rather than borrowing one designed for another model, directly controls sequence length, vocabulary coverage, and how the model sees numbers, code, and non-English text.
- 16 Vocabulary Size Trade-offs Choosing a tokeniser vocabulary size forces a three-way tension between sequence length, embedding table memory, and coverage of rare or multilingual text.
- 17 Code Data Curation A systematic account of how raw source code from the internet is transformed into a deduplicated, filtered, mixed, and decontaminated pretraining corpus for code-focused language models.
- 18 Data Mixtures and Domain Weighting Domain weighting determines how much of each data source a model sees during pretraining, and getting this wrong can cost tens of thousands of GPU-hours or silently cripple downstream task performance.
- 19 Data Pipelines at Scale Building a pretraining corpus requires extracting, filtering, deduplicating, and mixing hundreds of billions of tokens from heterogeneous sources while keeping benchmark contamination out.
- 20 Web-Scale Corpus Construction Building a pretraining corpus at web scale requires five tightly coupled stages - extraction, quality filtering, deduplication, source mixing, and decontamination - each of which silently determines what a model can and cannot know.
Synthetic Data Self-Instruct, distillation, self-play, and how to generate training data without collapsing. 20 concepts · 140 cards
- 01 Why Synthetic Data Synthetic data lets practitioners generate arbitrarily large, precisely labelled training sets without manual annotation, but only when the generator is accurate enough not to amplify its own errors.
- 02 Auditing Synthetic Data Auditing synthetic training data means applying measurable quality checks, coverage tests, and contamination scans before the data ever touches a training run.
- 03 Distillation and Terms-of-Service Constraints Using a commercial API to generate training data for a competing model almost universally violates the provider's terms of service, and understanding exactly why - and what compliant alternatives exist - is non-negotiable before building any distillation pipeline.
- 04 Distilling Reasoning Traces Reasoning-trace distillation transfers step-by-step chain-of-thought outputs from a large teacher model into fine-tuning data for a smaller student, giving the student reasoning capability it could not develop on final-answer supervision alone.
- 05 Diversity Metrics and Collapse Detection Quantitative metrics for detecting when synthetic training data loses coverage of the original distribution, with practical monitoring strategies to catch collapse before it harms the next model generation.
- 06 Evol-Instruct Evol-Instruct is an LLM-driven pipeline that iteratively rewrites seed instructions into progressively harder variants, enabling automated construction of complex instruction-following training data without human labellers.
- 07 Knowledge Distillation from a Teacher Model Knowledge distillation trains a smaller student model to reproduce a larger teacher's output distribution, enabling compact models with performance well beyond what their size alone would predict.
- 08 Mixing Synthetic and Human Data Balancing synthetic and human-authored training examples determines whether a model inherits the strengths of both sources or the worst of each.
- 09 Persona Prompting for Diversity Persona prompting injects fictional user identities into an LLM's prompt to steer it toward generating training data that spans a broader slice of the real distribution than naive repeated sampling achieves.
- 10 Programmatic and Templated Data Generation Programmatic and templated generation produces synthetic training data through deterministic code, slot-filling templates, and context-free grammars, giving precise control over distribution and format that purely model-driven pipelines cannot match.
- 11 Quality Filtering of Synthetic Data Quality filtering removes low-signal, redundant, or contaminated examples from synthetically generated datasets before fine-tuning, preventing reward hacking and model collapse.
- 12 Rejection Sampling Fine-Tuning and STaR Rejection sampling fine-tuning and the STaR algorithm are iterative self-improvement techniques that generate synthetic training data by having a model solve problems, keeping only the correct solutions, and re-training on those filtered traces.
- 13 Self-Instruct Self-Instruct is a bootstrapping pipeline that uses a language model's own outputs to generate large-scale instruction-following training data with minimal human annotation.
- 14 Self-Play and Self-Improvement Self-play loops use a model to generate, critique, and filter its own training data, compounding capability without proportionally scaling human annotation effort.
- 15 Synthetic Data for Code Generating synthetic code training data via instruction synthesis, distillation, and execution-based filtering lets small models punch well above their weight, but only when a reliable verifier anchors the loop.
- 16 Synthetic Data for Mathematics Synthetic math data pipelines use teacher models, rejection sampling, and question rewriting to bootstrap training corpora far larger than any human-labelled set, but the quality ceiling is set by the generator's own reasoning ability.
- 17 Synthetic Preference Data Synthetic preference data replaces expensive human comparison labels by having a language model judge which of two candidate responses is better, enabling scalable RLHF-style alignment without a large human labelling workforce.
- 18 Textbook-Quality Synthetic Data How to generate training data that teaches models to reason rather than memorise, using instruction synthesis, distillation, rejection sampling, and constitutional loops, along with the collapse risks that follow.
- 19 Model Collapse from Recursive Training When a model is trained repeatedly on its own outputs, tail distributions erode and the model progressively forgets rare but important knowledge, eventually producing impoverished, homogenised text.
- 20 The Constitutional AI Data Loop Constitutional AI replaces most human preference labels with a self-critique-and-revise loop guided by a written list of principles, producing both supervised fine-tuning data and AI-labelled preference pairs that train a reward model.
Training Dynamics & Scaling Learning-rate schedules, warmup, loss spikes, critical batch size, muP, and scaling laws. 11 concepts · 53 cards
- 01 Gradient Accumulation The trick that lets a single GPU simulate a batch size far larger than what fits in its memory, at the cost of wall-clock time rather than compute.
- 02 Gradient Clipping A five-character config value, clip norm 1.0, appears in nearly every published pretraining recipe, and it is a crude safety net rather than a fix, one that is easy to misread as more protective than it is.
- 03 Learning-Rate Schedules The shape of the learning-rate curve across a training run matters as much as its peak value, and getting the shape wrong wastes a slice of a compute budget that was never coming back.
- 04 Optimiser State and Memory A model's raw weight size is only the starting point for training memory; Adam alone roughly quadruples the footprint before a single activation has been stored.
- 05 Warmup and Why It Helps The first few hundred to few thousand steps of a large training run are its most fragile, and a short linear ramp on the learning rate is the cheapest insurance against wrecking them.
- 06 Weight Decay in Pretraining Pretraining rarely revisits the same token twice, so the classical overfitting story for weight decay barely applies, yet nearly every LLM recipe still sets it to a nonzero value close to 0.1.
- 07 Loss Spikes and Divergence On almost every long pretraining run the smooth descending loss curve suddenly lurches upward, and diagnosing and recovering from these spikes is still closer to an operational skill than a solved engineering problem.
- 08 Scaling Laws and the Chinchilla Correction How loss falls predictably with compute, parameters, and data, and why the Chinchilla result showed almost every large model of its era was badly undertrained.
- 09 The Critical Batch Size Past a point that can be measured but not derived from first principles, adding more GPUs to widen the batch stops buying faster convergence per token, and just burns compute for redundant gradient information.
- 10 Warmup-Stable-Decay Schedules Cosine decay bakes the total step count into its formula before training starts; warmup-stable-decay schedules split the learning-rate curve so that commitment can be made at the very end instead.
- 11 muP and Hyperparameter Transfer The learning rate that is optimal for a 40-million-parameter proxy model is usually wrong for the 70-billion-parameter model it was meant to stand in for, and muP is the parametrisation designed to make that transfer actually work.
Distributed Training Data, tensor and pipeline parallelism, ZeRO/FSDP sharding, mixed precision, and offload. 10 concepts · 70 cards
- 01 Data Parallelism and DDP How replicating the model and sharding the batch across GPUs scales training, and why AllReduce is the primitive every framework eventually depends on.
- 02 Fault Tolerance at Scale Why a 16,000-GPU run fails every few hours, how checkpoint interval trades wasted compute against write cost, and what asynchronous and in-memory checkpointing changed.
- 03 Gradient Checkpointing, Activation Recomputation, and CPU Offload Why activations - not weights - usually dominate training memory, and how recomputation and CPU/NVMe offload trade compute and bandwidth to fit larger models.
- 04 Mixed-Precision Training (FP16, BF16, FP8) How lower-precision formats halve memory and double throughput on tensor cores, why BF16 displaced FP16 for training, and what FP8 changes on H100 and Blackwell.
- 05 Composing Parallelism Strategies How data, tensor, pipeline, context and expert parallelism combine into one device mesh, why the ordering follows the interconnect hierarchy, and what each axis costs.
- 06 Overlapping Communication and Computation Why a large training run's collectives are mostly free when overlapped and catastrophic when exposed, and the bucketing, prefetch and scheduling tricks that hide them.
- 07 Pipeline Bubbles and Schedules Why pipeline parallelism wastes device time by construction, how the bubble fraction depends on micro-batch count, and what 1F1B, interleaving and zero-bubble schedules recover.
- 08 Sequence and Context Parallelism Why long-context training runs out of memory even when the model fits, and how Ring Attention and Ulysses split the sequence dimension across devices without changing the maths.
- 09 Tensor and Pipeline Parallelism How frontier labs split a model across thousands of GPUs by sharding within layers (tensor parallel) and across layers (pipeline parallel), and how to pick the split.
- 10 ZeRO and FSDP How sharding optimiser state, gradients, and parameters across data-parallel ranks turns a memory problem into a bandwidth problem, and why FSDP is now the PyTorch default.
Parameter-Efficient Fine-Tuning LoRA, QLoRA, DoRA, adapters and soft prompts — adapting big models on small budgets. 20 concepts · 140 cards
- 01 Adapter Modules Adapter modules insert small trainable bottleneck layers into a frozen pretrained transformer, achieving near-full fine-tuning performance while updating fewer than 4% of the model's parameters.
- 02 Catastrophic Forgetting in Fine-Tuning Full fine-tuning on a narrow task silently destroys general capabilities baked in during pre-training, and understanding why this happens is the prerequisite for choosing any mitigation strategy.
- 03 Choosing LoRA Rank and Alpha LoRA rank r controls how much task-specific capacity the adapter has, and alpha controls the scaling of that update; choosing them poorly wastes parameters or destabilises training.
- 04 IA3 and Scaling-Vector Methods IA3 fine-tunes a transformer by learning one scaling vector per targeted activation stream, achieving roughly 0.01% trainable parameters while matching full fine-tuning accuracy on several benchmarks.
- 05 LoRA Training Pitfalls LoRA's low-rank approximation introduces subtle failure modes around rank selection, learning rate asymmetry, and target-module coverage that can silently degrade fine-tuned model quality.
- 06 LoRA vs Full Fine-Tuning LoRA constrains weight updates to low-rank matrices, cutting trainable parameters by orders of magnitude while matching full fine-tuning quality on most tasks, but that constraint is also the source of its failure modes.
- 07 LoRA: Low-Rank Adaptation LoRA freezes a pretrained model's weights and inserts trainable low-rank matrix pairs into each target layer, cutting trainable parameters by up to 10,000x with no added inference latency.
- 08 Merging LoRA into Base Weights After LoRA training, the low-rank adapter matrices can be folded directly into the frozen base weights, eliminating inference overhead and adapter management complexity while producing a standard dense model.
- 09 Prefix Tuning Prefix tuning freezes all pretrained model weights and instead optimises a small set of continuous, task-specific vectors prepended to every layer's key-value cache, achieving within a few points of full fine-tuning while training roughly 0.1% of the original parameters.
- 10 Prompt Tuning with Soft Prompts Soft-prompt tuning prepends a small set of learnable continuous vectors to the input of a frozen language model, achieving full-fine-tuning parity at billion-parameter scale while touching less than 0.1% of model weights.
- 11 QLoRA: 4-bit Base with LoRA QLoRA reduces the GPU memory needed to fine-tune a 65-billion-parameter model from hundreds of gigabytes to a single 48 GB card by storing the frozen base model in 4-bit precision and training only small LoRA adapters at full 16-bit precision.
- 12 The Intrinsic-Dimension Hypothesis Pre-trained language models can be fine-tuned in a surprisingly low-dimensional subspace of their parameter space, and measuring that dimension explains why parameter-efficient methods work at all.
- 13 The Memory Maths of Fine-Tuning Fine-tuning a 7B-parameter model with full gradients and Adam state consumes roughly 112 GB of GPU memory; parameter-efficient methods cut that by an order of magnitude by training only a tiny fraction of weights.
- 14 Which Layers to Adapt Choosing which transformer weight matrices to inject LoRA or adapter modules into determines both the parameter budget and downstream task quality, and the right choice is not self-evident.
- 15 Why Parameter-Efficient Fine-Tuning Parameter-efficient fine-tuning methods adapt large pretrained models to new tasks by training only a small fraction of parameters, making customisation practical without the compute and storage costs of full fine-tuning.
- 16 Composing and Stacking Adapters Multiple trained adapters can be combined sequentially, by weighted sum, or through attention-based gating to build new capabilities without retraining the base model.
- 17 DoRA: Weight-Decomposed LoRA DoRA decomposes pre-trained weights into magnitude and direction components, then applies LoRA exclusively to the directional part, closing most of the accuracy gap between LoRA and full fine-tuning without adding inference overhead.
- 18 LoRA for Long-Context Adaptation Extending a model's context window via LoRA requires coordinating low-rank weight updates with position-encoding rescaling, and ignoring either side reliably degrades performance on long sequences.
- 19 NF4 and Double Quantisation NF4 is a 4-bit data type matched to the normal distribution of pretrained weights, and double quantisation further compresses the quantisation constants themselves, together enabling 65B-parameter models to fine-tune on a single 48 GB GPU via QLoRA.
- 20 Serving Many LoRA Adapters How specialised inference systems batch requests across hundreds of distinct LoRA adapters without duplicating the base model weights on GPU.
Alignment & Post-Training SFT, reward modelling, DPO/IPO/KTO/ORPO, model merging, and evaluating an aligned model. 21 concepts · 144 cards
- 01 Building a Preference Dataset A preference dataset pairs model outputs and records which one a human (or AI judge) preferred, providing the training signal that separates a helpful assistant from a raw base model.
- 02 Chat Templates and Special Tokens Chat templates are Jinja2 strings stored in a tokeniser that convert structured message lists into the exact token sequences a fine-tuned model was trained to process.
- 03 DPO in Practice DPO eliminates the separate reward model and RL loop of classic RLHF by reparameterising the reward directly into a classification loss over preferred and rejected response pairs.
- 04 Evaluating an Aligned Model Evaluating an aligned model requires measuring three partially competing properties simultaneously: helpfulness, harmlessness, and honesty, and every method for doing so introduces its own systematic biases.
- 05 KTO: Unpaired Preference Learning KTO aligns language models using only binary good/bad labels per response, avoiding the paired (chosen, rejected) format that makes preference data expensive and brittle to collect.
- 06 Length Bias and Verbosity Control Reward models trained on human preference data systematically score longer outputs higher regardless of quality, causing RLHF-trained models to inflate response length rather than improve content.
- 07 Model Merging: Linear and SLERP Linear and SLERP merging combine the weight tensors of separately fine-tuned models into a single deployable checkpoint, trading off alignment and capability at zero inference cost.
- 08 Model Soups Model soups average the weights of multiple independently fine-tuned checkpoints to produce a single model that outperforms any individual checkpoint without increasing inference cost.
- 09 ORPO and Reference-Free Alignment ORPO collapses supervised fine-tuning and preference alignment into a single training phase by appending a log-odds-ratio penalty directly to the NLL loss, removing the need for a reference model.
- 10 RLAIF and Constitutional Feedback RLAIF replaces human preference labels with an AI judge, and Constitutional AI extends this by encoding a written list of principles so the model critiques and revises its own outputs before any RL training begins.
- 11 Reward Hacking in Alignment Reward hacking occurs when a model maximises its training reward signal through behaviours that violate the designer's intent, undermining alignment despite high measured scores.
- 12 Reward Modelling in Practice A reward model is a learned surrogate for human preference that RLHF uses to provide a differentiable training signal, and its quality determines how aligned the final policy is.
- 13 Supervised Fine-Tuning for Instructions Supervised fine-tuning on curated instruction-response pairs is the first step that transforms a raw pretrained language model into a model that reliably follows human instructions.
- 14 The KL Penalty and Reference Model The KL penalty constrains a fine-tuned language model to stay statistically close to its pre-trained reference, preventing reward hacking while preserving the capabilities built during pretraining.
- 15 The Post-Training Pipeline A structured walkthrough of the four-stage process that converts a raw pretrained language model into a deployable assistant, from supervised fine-tuning through reward modelling, RLHF, and preference-optimisation alternatives.
- 16 DPO and Preference Optimisation How Direct Preference Optimisation collapses the reward-model-plus-PPO pipeline into a single classification loss, and where the RLHF machinery still earns its keep.
- 17 IPO and the Overfitting Fix IPO replaces DPO's sigmoid loss with a squared identity transform, eliminating the theoretical overfitting guarantee that breaks when preference data is finite and deterministic.
- 18 Multi-Objective Alignment Multi-objective alignment trains a single language model to satisfy several competing criteria simultaneously by navigating the Pareto front of reward trade-offs rather than collapsing them into one scalar.
- 19 Online vs Offline Preference Optimisation Offline preference optimisation trains on a fixed dataset of ranked responses, while online methods continuously sample from the current policy, and that single difference has substantial consequences for distribution coverage, reward hacking risk, and final alignment quality.
- 20 PPO for RLHF in Practice A concrete walkthrough of how Proximal Policy Optimisation is wired into the RLHF pipeline, covering the four-model setup, the clipped objective, KL penalty shaping, and the failure modes that kill real training runs.
- 21 TIES and DARE Merging TIES and DARE are two parameter-space merging algorithms that resolve weight interference when combining multiple fine-tuned models into one, avoiding retraining entirely.