Concept library
1134 concepts across 20 domains and 101 tracks. Each track is a coherent sequence — read it top to bottom or dip in wherever the gap is.
All domains
01Foundations
02Transformer Internals
03Training & Fine-Tuning
04Reinforcement Learning
05Inference, Systems & Hardware
06Applied LLM Engineering
07Reasoning, Evaluation & Safety
08Multimodal & Applications
09Classical ML & Statistical Learning
10Causal Inference & Experimentation
11Time Series & Forecasting
12Graphs, Recommenders & Structured Data
13Generative Modelling Beyond Transformers
14Efficiency, Compression & Edge AI
15Search & Information Retrieval
16Data & Feature Engineering
17MLOps & Platform Engineering
18Security, Privacy & Adversarial ML
19Governance, Risk & Responsible AI
20Human-AI Interaction, Product & Economics
17
MLOps & Platform Engineering
Everything between a notebook that works and a system that keeps working.
6tracks
48concepts
485cards
5.8hreading
Experiment Tracking & Reproducibility Run metadata, seeds and determinism, environment capture, and what it takes to rebuild a result a year later. 6 concepts · 67 cards
- 01 Environment Capture and Pinning Why a requirements file does not describe an environment, the layers below Python that also move, and the tradeoff between reproducibility and being able to patch a vulnerability.
- 02 Run Metadata and What to Record The minimum set of facts that makes a training run comparable and rebuildable a year later, why metrics are the least important part, and the discipline that makes tracking survive contact with a deadline.
- 03 Data Versioning and Content Addressing Why versioning datasets is harder than versioning code, how content addressing makes it tractable without copying, and what a dataset version has to mean to be useful.
- 04 Hyperparameter Search Hygiene Why random search beats grid search, how early-stopping schedulers change the budget calculation, and the selection bias that makes the best run's reported score an overestimate.
- 05 Multi-Seed Reporting and Run-to-Run Variance How many seeds a comparison needs, which sources of variance a seed sweep should randomise, and the reporting conventions that stop a one-run improvement from being mistaken for a result.
- 06 Seeds and the Limits of Determinism Why setting a seed does not make a training run reproducible, the specific sources of non-determinism on a GPU, and how to decide how much determinism is worth paying for.
Model Registry & Versioning Artefact lineage, promotion gates, rollback, and versioning models and their data together. 6 concepts · 67 cards
- 01 Model Cards and Intended Use What documentation a model version needs to be safely reused by someone who did not build it, why disaggregated evaluation is the substantive part, and how model cards become checkbox exercises.
- 02 The Registry as a Promotion Gate Why a model registry is a workflow rather than a storage location, what a stage transition should require, and the difference between a registry that records decisions and one that enforces them.
- 03 Model Lineage Across Retraining Why a chain of fine-tuned and distilled models makes provenance a graph rather than a record, what has to be tracked at each edge, and the failures that follow from losing it.
- 04 Rollback, Pinning and Reproducible Serving What it takes to revert a model change under incident conditions, why the model is usually not the only thing that has to revert, and the mechanisms that make a rollback fast rather than heroic.
- 05 Versioning Compound LLM Systems Why an LLM application's version must bind the model snapshot, prompt, retrieval index, tools and decoding settings into one identifier, and how provider deprecations turn rollback from a button into a migration.
- 06 What a Model Version Actually Contains Why weights alone are not a deployable model, the full set of artefacts that must move together, and the coupling failures that occur when one of them is versioned separately.
CI/CD for ML Testing pipelines that emit models, data-dependent tests, progressive delivery and automated retraining. 6 concepts · 69 cards
- 01 Build Once, Promote the Artefact Why rebuilding per environment reintroduces every difference the pipeline was meant to eliminate, how configuration is separated from the immutable artefact, and where ML deployments break the pattern.
- 02 Testing Pipelines That Emit Models What can be tested deterministically in an ML codebase, why model quality is not a unit test, and the layered test strategy that keeps CI fast while still catching the failures that matter.
- 03 Automated Retraining and Its Triggers When a retraining loop is worth building, the four trigger designs and what each optimises, and the failure modes that make automated retraining actively dangerous.
- 04 Data-Dependent Tests and Behavioural Suites Why aggregate metrics are a poor gate, how curated behavioural cases catch regressions that averages hide, and the discipline of turning every production failure into a permanent test.
- 05 LLM Regression Testing in CI How to gate pull requests on a stochastic system, covering golden-set sizing, paired comparisons, judge variance, flaky thresholds and the per-PR cost budget that decides what runs where.
- 06 Progressive Delivery for Models The staged rollout patterns that separate correctness validation from quality validation, why shadow deployment answers a question canary cannot, and what each stage should be watching.
ML Observability & Drift Feature and prediction monitoring, delayed labels, drift statistics, and alerting that does not cry wolf. 11 concepts · 101 cards
- 01 Delayed labels and deferred evaluation What to do when ground truth for today's predictions arrives in weeks or months, how label lag distorts monitoring windows, and which proxies are safe to alert on.
- 02 Harmful versus benign shift Why a statistically significant change in the input distribution is neither necessary nor sufficient for a drop in model performance, and what to monitor instead.
- 03 Prediction Logging and Traceability What to record at inference so that a question asked three months later has an answer, why the feature vector matters more than the input, and the sampling and privacy tradeoffs.
- 04 The Three Drifts and How They Differ Covariate shift, label shift and concept drift decomposed precisely, why only one of them necessarily degrades a model, and which of them your monitoring can actually see.
- 05 Drift Statistics and What They Miss PSI, KL divergence, KS and MMD compared on what they detect and where they fail, why per-feature tests miss joint shifts, and the multiple-comparison problem that makes wide monitoring noisy.
- 06 Monitoring Without Labels What to watch when ground truth arrives months late or never, why prediction distributions and confidence are the highest-value proxies, and how to estimate performance from unlabelled data.
- 07 Observability for LLM Applications Why the classical monitoring stack does not transfer to systems with free-text output, what a trace over an agent must capture, and the online quality signals that work without ground truth.
- 08 Sequential monitoring and always-valid alarms Why running a fixed-sample test every day inflates false alarms, and how confidence sequences, e-values and conformal test martingales give bounds that stay valid under continuous peeking.
- 09 Shift attribution and incident triage Once a monitor fires, how to decompose the performance change across data quality, covariate shift and concept change, and the order of checks that resolves most model incidents fastest.
- 10 Slice-Based Monitoring and Alert Design Why aggregate model metrics hide failures on important subgroups, how sample size and multiple testing limit what per-slice monitoring can detect, and how to route slice alerts so they stay actionable.
- 11 Unlabelled accuracy estimation How confidence thresholds, confusion-matrix inversion and model disagreement estimate a deployed model's current error rate without any ground truth, and which assumption each one buys that estimate with.
GPU Fleet & Capacity Scheduling and quota, fragmentation, preemption, multi-tenancy, and planning capacity under lumpy demand. 7 concepts · 76 cards
- 01 Capacity Planning Under Lumpy Demand Why GPU demand does not smooth the way CPU demand does, how to reason about the reserved-versus-on-demand mix, and the lead times that make this a quarters-ahead decision.
- 02 Fault Tolerance for Long Training Runs Why failure is the expected case at scale, the arithmetic that sets checkpoint frequency, and the detection problem that makes silent corruption worse than a crash.
- 03 GPU Health Checks and Silent Data Corruption What the published interruption statistics from Llama 3, Gemini and Meta's research clusters say about hardware failure at scale, and how burn-in, periodic checks, straggler detection and SDC screening keep bad nodes out of synchronous jobs.
- 04 Gang Scheduling and Fragmentation Why a distributed training job cannot start until every worker starts, how that requirement produces both deadlock and stranded capacity, and the placement constraints that make a cluster's usable size smaller than its size.
- 05 Multi-Tenancy and GPU Sharing Why one job per GPU wastes most of the hardware for small models, the three sharing mechanisms and their isolation guarantees, and the interference that makes sharing unsuitable for latency-sensitive work.
- 06 Preemption, Priority and Spot Capacity How interruptible capacity changes the cost of training by a large factor, what a job must be able to do to use it, and the failure modes that make spot capacity a false economy for the unprepared.
- 07 Topology-Aware Scheduling How NVLink domains, rail-optimised fabrics and switch hierarchy make GPU placement a performance variable, the collective cost model that quantifies it, and how schedulers trade placement quality against queueing delay.
Cost & FinOps for AI Unit economics per request, token accounting, reserved versus spot capacity, and attributing spend to features. 12 concepts · 105 cards
- 01 Cost per Completed Task Why a list price per million tokens cannot rank models for a workload, how to compute cost per completed task instead, and the measured evidence that the cheaper model per token is often the more expensive one per outcome.
- 02 Non-Token Charges in an AI Bill The line items a cost-per-token model cannot see, covering per-call tool fees, sandbox and session runtime, residency premiums and the token overhead of tool definitions, and why they decide the bill for agentic workloads.
- 03 Per-Request Energy, Water and Carbon How to put an energy, water and carbon figure on a single inference request, why the two most rigorous published per-request figures differ by a factor of 38 on carbon and 173 on water, and which accounting choices produce the gap.
- 04 Reserved, On-Demand and the Shape of Commitment How to choose a commitment mix when demand is uncertain, why the break-even is simply a price ratio, and the option value that makes shorter commitments rational despite costing more.
- 05 Self-Hosting Versus API Break-Even A worked break-even model for serving an open-weights model yourself against paying per token, showing why utilisation, not the hourly GPU price, usually decides the answer, and which assumptions flip it.
- 06 Spend Guardrails and Quotas Why AI spend can rise by orders of magnitude in hours, the layered controls that bound it without blocking legitimate work, and the design of a kill switch that is actually usable.
- 07 The Price Menu: Caching, Batch and Service Tiers The multipliers that sit between a model's list price and what you actually pay, the break-even arithmetic for a cache write, and why the menu exists as price discrimination rather than generosity.
- 08 Unit Economics of an AI Feature How to build a cost-per-request figure that survives scrutiny, why the marginal cost of an LLM feature does not fall with scale the way software's does, and the retry and failure multipliers everyone forgets.
- 09 Cost Variance and the Tail of Agentic Spend Why the budget for an agent workload is set by the tail of its cost distribution rather than its mean, where the multipliers in that tail come from, and the controls that bound a single run without quietly lowering success.
- 10 The Cost of Evaluation and Experimentation Why evaluation is a standing line item that can rival production inference, the multipliers that make a golden set expensive, and how to buy statistical confidence at the lowest price.
- 11 Token Accounting and Cost Attribution Why a single provider invoice cannot be allocated to teams, features or customers without instrumentation, and the tagging discipline that makes AI spend attributable.
- 12 Training Cost Estimation Before You Commit The FLOP arithmetic that converts a model and dataset size into a GPU-hour figure, why achieved utilisation is the term that decides the answer, and the overheads that turn an estimate into a budget.