Concept library
1015 concepts across 20 domains and 101 tracks. Each track is a coherent sequence — read it top to bottom or dip in wherever the gap is.
All domains
01Foundations
02Transformer Internals
03Training & Fine-Tuning
04Reinforcement Learning
05Inference, Systems & Hardware
06Applied LLM Engineering
07Reasoning, Evaluation & Safety
08Multimodal & Applications
09Classical ML & Statistical Learning
10Causal Inference & Experimentation
11Time Series & Forecasting
12Graphs, Recommenders & Structured Data
13Generative Modelling Beyond Transformers
14Efficiency, Compression & Edge AI
15Search & Information Retrieval
16Data & Feature Engineering
17MLOps & Platform Engineering
18Security, Privacy & Adversarial ML
19Governance, Risk & Responsible AI
20Human-AI Interaction, Product & Economics
08
Multimodal & Applications
Beyond text — vision, speech, robotics and scientific discovery.
5tracks
74concepts
669cards
9.8hreading
Vision & Multimodal ViT, CLIP, diffusion, SAM, and the vision-language models that read images as tokens. 12 concepts · 131 cards
- 01 Contrastive Vision-Language: CLIP How a 400M image-text contrastive objective produced a shared embedding space that does zero-shot classification, retrieval, and grounding without any task-specific labels.
- 02 Multimodal Hallucination and Benchmark Validity Why vision-language models describe objects that are not in the image, and why a large share of multimodal benchmark scores can be reproduced without showing the model any image at all.
- 03 Native-Resolution Vision Encoding Why squashing every image to a fixed square destroyed text legibility in early VLMs, and how tiling, patch packing and dynamic resolution replaced it at the cost of an unbounded visual token budget.
- 04 Segment Anything (SAM) and Dense Prediction How promptable segmentation became a foundation-model task, what SAM's encoder-decoder split was designed for, and where it still loses to specialist models.
- 05 Video, Audio, and Any-to-Any Models How Whisper, V-JEPA, Sora-class video generators, MusicGen, and unified any-to-any models extend the multimodal stack beyond static images.
- 06 Vision Transformers (ViT) How treating an image as a sequence of patches let pure transformers beat CNNs once data crossed the 300M-image mark, and what the architecture gave up to get there.
- 07 Visual Document Retrieval Why the OCR-parse-chunk-embed pipeline loses exactly the documents that matter, and how late-interaction retrieval over page images replaces five brittle stages with one model.
- 08 Diffusion Models How learning to invert a noise process became the dominant generative recipe for images, video, and audio, and why Flow Matching and DiTs are reshaping the recipe in 2024.
- 09 Discrete Visual Tokenisers How VQ-VAE, VQGAN and FSQ turn an image into a short sequence of integers, why codebook collapse wrecks half of them, and what the compression ratio costs in reconstruction fidelity.
- 10 Early-Fusion Mixed-Modal Models The difference between bolting a vision encoder onto a frozen LLM and training one transformer over interleaved image and text tokens from scratch, and why the second approach destabilises training.
- 11 Flow Matching and Rectified Flow Why regressing a velocity field along straight noise-to-data paths replaced score matching in frontier image models, and what the straight-line claim does and does not guarantee.
- 12 Multimodal LLMs: LLaVA, Flamingo, GPT-4V The vision-encoder-plus-projector-plus-LLM recipe that dominates open multimodal models, why Flamingo's perceiver design still matters for video, and what native-multimodal frontier models do differently.
Speech Recognition Spectrograms, CTC, RNN-T, Conformer, Whisper, streaming, diarisation and self-supervised audio. 20 concepts · 140 cards
- 01 Audio Features and Spectrograms Raw audio waveforms are rarely fed directly to speech models; this concept explains how and why they are first converted into spectrogram-based representations that compress perceptual information into a learnable 2-D grid.
- 02 The ASR Problem and Pipeline Automatic speech recognition converts a raw audio waveform into a word sequence by solving an alignment problem that classical NLP never had to face.
- 03 Beam Search Decoding in ASR Beam search decoding navigates the exponentially large label sequence space in ASR by maintaining a fixed-width frontier of the most probable partial hypotheses at each step, making it the default inference strategy for CTC, RNN-T, and attention-based models.
- 04 CTC: Connectionist Temporal Classification CTC is a training objective that lets a neural network learn to align variable-length audio to text without any hand-labelled frame-level annotations.
- 05 Causal and Chunked Attention for Streaming Streaming ASR requires attention mechanisms that never look at future audio; causal masking and chunked attention are the two principal techniques, each trading latency against accuracy in different ways.
- 06 Endpointing and Voice Activity Detection Endpointing and voice activity detection are the mechanisms that decide when a user has finished speaking, directly controlling the latency and correctness of every streaming ASR system.
- 07 Listen, Attend and Spell LAS is a purely sequence-to-sequence ASR model that replaces HMMs, CTC, and explicit pronunciation dictionaries with a pyramidal RNN encoder and an attention-based character decoder trained end-to-end.
- 08 Robustness to Noise and Accents How modern ASR systems are trained and adapted to handle environmental noise, channel distortions, and speaker accent variability without collapsing to near-zero accuracy.
- 09 Speaker Diarisation Speaker diarisation segments an audio recording into speaker-homogeneous regions and assigns each region a speaker identity, answering the question "who spoke when" without necessarily transcribing what was said.
- 10 Streaming vs Offline ASR Streaming ASR emits partial transcripts incrementally as audio arrives, trading access to future context for low latency, while offline ASR processes the full utterance and consistently achieves lower word error rates.
- 11 The CTC Blank Token and Alignment The CTC blank token is a special output symbol that lets a neural network emit one label per time-step without needing a hand-crafted alignment between audio frames and characters.
- 12 The Conformer Architecture The Conformer interleaves convolution and multi-head self-attention inside each encoder block to capture both fine-grained local acoustic patterns and long-range sequence dependencies, achieving state-of-the-art ASR accuracy on LibriSpeech.
- 13 Whisper's Multitask Decoder Whisper conditions a single sequence-to-sequence decoder on a prefix of special tokens that specify language, task, and timestamp behaviour, allowing one model to handle transcription, translation, and language identification without any task-specific heads.
- 14 Whisper: Weakly-Supervised ASR Whisper trains a sequence-to-sequence Transformer on 680,000 hours of weakly-supervised internet audio to achieve robust multilingual speech recognition without task-specific fine-tuning.
- 15 Why Sequence Length Is Hard in Audio Audio produces roughly 100 frames per second of speech, making sequence lengths 10-50x longer than equivalent text, and this mismatch drives almost every architectural and training decision in modern ASR.
- 16 Word Error Rate and ASR Evaluation Word Error Rate measures ASR accuracy as the minimum edit distance between a hypothesis and a reference transcript, normalised by reference length, but its apparent simplicity hides a nest of normalisation choices that make numbers across papers routinely incomparable.
- 17 wav2vec 2.0 and Self-Supervised Audio wav2vec 2.0 learns speech representations from raw audio without transcripts by masking latent features and solving a contrastive task over learned discrete units, then fine-tunes on as little as ten minutes of labelled speech to reach competitive word error rates.
- 18 HuBERT and Discrete Audio Units HuBERT pre-trains a speech encoder by predicting offline k-means cluster labels for masked audio frames, producing discrete unit sequences that rival phoneme transcriptions without any text supervision.
- 19 Language-Model Fusion Language-model fusion techniques inject text-only knowledge into end-to-end ASR models at inference time or training time, and the correct method depends on how much implicit language bias the acoustic model has already absorbed.
- 20 The RNN-Transducer The RNN-Transducer is a fully neural, streaming-capable sequence transduction model that replaces CTC's conditional independence assumption with a learned label-context network, enabling accurate on-device speech recognition.
Speech Synthesis Acoustic models and vocoders, Tacotron, FastSpeech, HiFi-GAN, neural codecs and voice cloning. 20 concepts · 139 cards
- 01 The TTS Problem and Pipeline Text-to-speech converts a string of characters into a waveform through a chain of normalisation, acoustic modelling, and synthesis stages, each introducing its own failure modes.
- 02 Attention Failures in TTS Attention-based TTS systems fail in predictable ways - word skipping, repetition, and unstable alignment - and understanding the mechanics behind each failure mode is essential for building reliable speech synthesis pipelines.
- 03 Bark and Fully Generative Audio Bark is a transformer-based model that generates speech, music, and nonverbal audio from text by autoregressively predicting discrete audio codec tokens, without any phoneme pipeline or continuous acoustic model.
- 04 Duration Modelling and Alignment Duration modelling assigns how many audio frames each phoneme occupies; alignment is the mechanism that learns or infers that mapping from text-audio pairs without manual annotation.
- 05 Evaluating TTS with MOS Mean Opinion Score is the field's primary yardstick for TTS naturalness, but its reliability hinges on listener pool design, anchoring, and context choices that most papers under-report.
- 06 FastSpeech and Non-Autoregressive TTS FastSpeech eliminates the sequential mel-frame dependency of autoregressive TTS by predicting all frames in parallel, trading model simplicity for a duration predictor and a length regulator.
- 07 Griffin-Lim and Classical Vocoders Griffin-Lim is an iterative phase-recovery algorithm that converts a magnitude spectrogram back into a waveform without any learned parameters, and understanding it clarifies exactly what neural vocoders had to replace and why.
- 08 HiFi-GAN and Neural Vocoders Neural vocoders convert acoustic feature representations into raw audio waveforms, and HiFi-GAN achieves near-human quality at 167x real-time speed using a multi-scale, multi-period GAN architecture.
- 09 Neural Audio Codecs Neural audio codecs compress waveforms into discrete token sequences using learned vector quantisation, enabling language models to generate speech token-by-token.
- 10 Prosody and Style Control How TTS systems encode and manipulate pitch, duration, energy, and speaking style so that synthesised speech sounds intended rather than merely intelligible.
- 11 Residual Vector Quantisation Residual vector quantisation stacks multiple codebooks to approximate a continuous audio embedding with increasingly fine-grained corrections, making it the compression backbone of modern neural audio codecs.
- 12 Streaming TTS and Latency Streaming TTS pipelines generate and deliver audio incrementally to cut time-to-first-audio from several seconds to under 300 ms, but doing so imposes hard trade-offs on chunk size, model architecture, and prosodic coherence.
- 13 Tacotron 2 Tacotron 2 is a two-stage neural TTS pipeline that converts text to mel spectrograms with a sequence-to-sequence model, then synthesises raw audio with a conditioned WaveNet vocoder, achieving near-human MOS scores.
- 14 Text Normalisation and Phonemisation Text normalisation converts raw written text into a speakable form, and phonemisation maps those words to phoneme sequences; together they determine what a TTS system says before any audio is generated.
- 15 The Acoustic Model and Vocoder Split Modern neural TTS splits the problem into two specialised sub-networks - an acoustic model that maps text to a compact spectral representation, and a vocoder that reconstructs a full audio waveform from that representation.
- 16 The Mel-Spectrogram Interface The mel spectrogram is the agreed-upon intermediate representation that decouples acoustic modelling from waveform generation in modern TTS pipelines.
- 17 WaveNet WaveNet is a fully autoregressive convolutional neural network that models raw audio waveforms one sample at a time, achieving near-human speech quality at the cost of extremely slow sequential generation.
- 18 Diffusion Models for Speech Diffusion models iteratively denoise random Gaussian noise into speech waveforms or mel-spectrograms, achieving sample quality that matches autoregressive vocoders at a fraction of the sequential compute cost.
- 19 VALL-E: TTS as Token Language Modelling VALL-E reformulates text-to-speech as a conditional language modelling problem over discrete audio codec tokens, enabling zero-shot voice cloning from a three-second recording by treating acoustic context the same way GPT treats a few-shot text prompt.
- 20 Zero-Shot Voice Cloning Zero-shot voice cloning synthesises speech in the voice of an unseen speaker from a short reference recording, without any fine-tuning at inference time.
Robotics & Embodied AI Vision-language-action models, action tokenisation, diffusion policies and sim-to-real. 10 concepts · 122 cards
- 01 Action Tokenisation and Representation How continuous robot actions become discrete tokens an autoregressive transformer can emit, from per-dimension binning to frequency-space compression.
- 02 Evaluating Robot Policies Why a reported 70 percent success rate on a robot task usually carries an error bar of plus or minus 15 points, and what protocol changes make two policies actually comparable.
- 03 Imitation Learning and Diffusion Policies Why cloning a demonstrator's actions drifts into unseen states, and how generative action models such as diffusion policies and action chunking control the drift.
- 04 Massively Parallel Simulation and Sim-to-Real Locomotion How putting the physics engine on the GPU turned weeks of legged-robot RL into minutes, and why domain randomisation is the only reason those policies survive contact with a real floor.
- 05 Sim-to-Real Transfer Why robot policies are trained in simulation, why they break on real hardware, and how domain randomisation closes the reality gap by making the real world look like one more random draw.
- 06 Cross-Embodiment Transfer Why robot data has never had an ImageNet, how Open X-Embodiment pooled 22 robot types into one training set, and what has to be true for data from one arm to help a different arm.
- 07 Flow-Matching Action Experts Why binning continuous robot actions into text tokens caps control frequency, and how attaching a flow-matching action head to a vision-language model reaches 50 Hz dexterous control.
- 08 RT-2 and Web-Scale Robot Learning RT-2 co-trains one transformer on internet vision-language data and robot trajectories by encoding actions as text tokens, transferring semantic web knowledge into robotic control.
- 09 Vision-Language-Action Models Turning a pretrained vision-language model into a robot policy that maps camera images plus a language instruction to motor actions, so the robot inherits web-scale semantic knowledge it could never learn from robot data alone.
- 10 World Models for Control How learning a latent dynamics model lets an agent train inside its own imagination, why that is the answer to robotics' sample-efficiency problem, and where the learned simulator's errors get exploited.
AI for Science AlphaFold, protein language models, materials discovery, and the pitfalls of ML-for-science. 12 concepts · 137 cards
- 01 Machine Learning for Materials Discovery How graph neural networks screen millions of candidate crystals for stability and how machine-learning interatomic potentials approximate DFT cheaply enough to simulate them, plus why an in-silico "stable" material is not yet a real one.
- 02 Pitfalls in ML-for-Science The failure modes, chiefly data leakage, that make machine-learning results in scientific papers look stronger than they replicate, and the reporting standards proposed to catch them.
- 03 Protein Language Models How masked-language-model pretraining over amino-acid sequences produces structure and function signal, and how ESMFold trades some accuracy for dropping the MSA search that AlphaFold2 depends on.
- 04 Self-Driving Labs and Autonomous Experimentation The A-Lab synthesised 41 of 58 target compounds in 17 days with no human intervention, and the dispute that followed — ending in a 2026 author correction — is the clearest available lesson in what an autonomous laboratory actually automates.
- 05 AI for Formal Mathematics How proof assistants turn mathematics into a verifiable reward signal, what AlphaGeometry and AlphaProof achieved at the IMO, and why autoformalisation remains the bottleneck.
- 06 AlphaFold2 and the Protein-Folding Problem How a deep-learning system read co-evolution signal out of aligned protein sequences to predict 3D structure at near-experimental accuracy, and what it still cannot do.
- 07 AlphaFold3 and Biomolecular Co-Folding How AlphaFold3 dropped the protein-only structure module for a diffusion head that denoises raw atoms, letting one model co-fold proteins with ligands, nucleic acids, ions, and modified residues, and how the open reimplementations caught up.
- 08 Generative Protein and Molecule Design Structure prediction reads nature's proteins; generative design writes new ones — how RFdiffusion denoises backbones into existence, why a separate network is needed to choose the sequence, and why the only benchmark that counts is a wet-lab success rate.
- 09 Machine-Learned Interatomic Potentials How equivariant graph networks reach near-quantum accuracy at a fraction of the cost, why symmetry is designed in rather than learned, and what breaks when a potential leaves its training chemistry.
- 10 Neural Operators and PDE Surrogates Why learning a mapping between function spaces is different from fitting a network to a grid, how the Fourier neural operator achieves resolution invariance, and what a surrogate cannot promise.
- 11 Neural Weather Prediction How graph and transformer models trained on reanalysis data overtook physics-based forecasting on most verification targets, what they still depend on, and where the learned approach genuinely fails.
- 12 Single-Cell Foundation Models Pretraining a transformer on tens of millions of single-cell transcriptomes produced scGPT and Geneformer, and then a zero-shot benchmark found both losing to highly variable gene selection — a case study in what "foundation model" does and does not transfer.