Concept library
446 concepts across 8 domains and 36 tracks. Each track is a coherent sequence — read it top to bottom or dip in wherever the gap is.
08
Multimodal & Applications
Beyond text — vision, speech, robotics and scientific discovery.
5tracks
60concepts
395cards
7.9hreading
Vision & Multimodal ViT, CLIP, diffusion, SAM, and the vision-language models that read images as tokens. 6 concepts · 32 cards
- 01 Contrastive Vision-Language: CLIP How a 400M image-text contrastive objective produced a shared embedding space that does zero-shot classification, retrieval, and grounding without any task-specific labels.
- 02 Segment Anything (SAM) and Dense Prediction How promptable segmentation became a foundation-model task, what SAM's encoder-decoder split was designed for, and where it still loses to specialist models.
- 03 Video, Audio, and Any-to-Any Models How Whisper, V-JEPA, Sora-class video generators, MusicGen, and unified any-to-any models extend the multimodal stack beyond static images.
- 04 Vision Transformers (ViT) How treating an image as a sequence of patches let pure transformers beat CNNs once data crossed the 300M-image mark, and what the architecture gave up to get there.
- 05 Diffusion Models How learning to invert a noise process became the dominant generative recipe for images, video, and audio, and why Flow Matching and DiTs are reshaping the recipe in 2024.
- 06 Multimodal LLMs: LLaVA, Flamingo, GPT-4V The vision-encoder-plus-projector-plus-LLM recipe that dominates open multimodal models, why Flamingo's perceiver design still matters for video, and what native-multimodal frontier models do differently.
Speech Recognition Spectrograms, CTC, RNN-T, Conformer, Whisper, streaming, diarisation and self-supervised audio. 20 concepts · 140 cards
- 01 Audio Features and Spectrograms Raw audio waveforms are rarely fed directly to speech models; this concept explains how and why they are first converted into spectrogram-based representations that compress perceptual information into a learnable 2-D grid.
- 02 The ASR Problem and Pipeline Automatic speech recognition converts a raw audio waveform into a word sequence by solving an alignment problem that classical NLP never had to face.
- 03 Beam Search Decoding in ASR Beam search decoding navigates the exponentially large label sequence space in ASR by maintaining a fixed-width frontier of the most probable partial hypotheses at each step, making it the default inference strategy for CTC, RNN-T, and attention-based models.
- 04 CTC: Connectionist Temporal Classification CTC is a training objective that lets a neural network learn to align variable-length audio to text without any hand-labelled frame-level annotations.
- 05 Causal and Chunked Attention for Streaming Streaming ASR requires attention mechanisms that never look at future audio; causal masking and chunked attention are the two principal techniques, each trading latency against accuracy in different ways.
- 06 Endpointing and Voice Activity Detection Endpointing and voice activity detection are the mechanisms that decide when a user has finished speaking, directly controlling the latency and correctness of every streaming ASR system.
- 07 Listen, Attend and Spell LAS is a purely sequence-to-sequence ASR model that replaces HMMs, CTC, and explicit pronunciation dictionaries with a pyramidal RNN encoder and an attention-based character decoder trained end-to-end.
- 08 Robustness to Noise and Accents How modern ASR systems are trained and adapted to handle environmental noise, channel distortions, and speaker accent variability without collapsing to near-zero accuracy.
- 09 Speaker Diarisation Speaker diarisation segments an audio recording into speaker-homogeneous regions and assigns each region a speaker identity, answering the question "who spoke when" without necessarily transcribing what was said.
- 10 Streaming vs Offline ASR Streaming ASR emits partial transcripts incrementally as audio arrives, trading access to future context for low latency, while offline ASR processes the full utterance and consistently achieves lower word error rates.
- 11 The CTC Blank Token and Alignment The CTC blank token is a special output symbol that lets a neural network emit one label per time-step without needing a hand-crafted alignment between audio frames and characters.
- 12 The Conformer Architecture The Conformer interleaves convolution and multi-head self-attention inside each encoder block to capture both fine-grained local acoustic patterns and long-range sequence dependencies, achieving state-of-the-art ASR accuracy on LibriSpeech.
- 13 Whisper's Multitask Decoder Whisper conditions a single sequence-to-sequence decoder on a prefix of special tokens that specify language, task, and timestamp behaviour, allowing one model to handle transcription, translation, and language identification without any task-specific heads.
- 14 Whisper: Weakly-Supervised ASR Whisper trains a sequence-to-sequence Transformer on 680,000 hours of weakly-supervised internet audio to achieve robust multilingual speech recognition without task-specific fine-tuning.
- 15 Why Sequence Length Is Hard in Audio Audio produces roughly 100 frames per second of speech, making sequence lengths 10-50x longer than equivalent text, and this mismatch drives almost every architectural and training decision in modern ASR.
- 16 Word Error Rate and ASR Evaluation Word Error Rate measures ASR accuracy as the minimum edit distance between a hypothesis and a reference transcript, normalised by reference length, but its apparent simplicity hides a nest of normalisation choices that make numbers across papers routinely incomparable.
- 17 wav2vec 2.0 and Self-Supervised Audio wav2vec 2.0 learns speech representations from raw audio without transcripts by masking latent features and solving a contrastive task over learned discrete units, then fine-tunes on as little as ten minutes of labelled speech to reach competitive word error rates.
- 18 HuBERT and Discrete Audio Units HuBERT pre-trains a speech encoder by predicting offline k-means cluster labels for masked audio frames, producing discrete unit sequences that rival phoneme transcriptions without any text supervision.
- 19 Language-Model Fusion Language-model fusion techniques inject text-only knowledge into end-to-end ASR models at inference time or training time, and the correct method depends on how much implicit language bias the acoustic model has already absorbed.
- 20 The RNN-Transducer The RNN-Transducer is a fully neural, streaming-capable sequence transduction model that replaces CTC's conditional independence assumption with a learned label-context network, enabling accurate on-device speech recognition.
Speech Synthesis Acoustic models and vocoders, Tacotron, FastSpeech, HiFi-GAN, neural codecs and voice cloning. 20 concepts · 139 cards
- 01 The TTS Problem and Pipeline Text-to-speech converts a string of characters into a waveform through a chain of normalisation, acoustic modelling, and synthesis stages, each introducing its own failure modes.
- 02 Attention Failures in TTS Attention-based TTS systems fail in predictable ways - word skipping, repetition, and unstable alignment - and understanding the mechanics behind each failure mode is essential for building reliable speech synthesis pipelines.
- 03 Bark and Fully Generative Audio Bark is a transformer-based model that generates speech, music, and nonverbal audio from text by autoregressively predicting discrete audio codec tokens, without any phoneme pipeline or continuous acoustic model.
- 04 Duration Modelling and Alignment Duration modelling assigns how many audio frames each phoneme occupies; alignment is the mechanism that learns or infers that mapping from text-audio pairs without manual annotation.
- 05 Evaluating TTS with MOS Mean Opinion Score is the field's primary yardstick for TTS naturalness, but its reliability hinges on listener pool design, anchoring, and context choices that most papers under-report.
- 06 FastSpeech and Non-Autoregressive TTS FastSpeech eliminates the sequential mel-frame dependency of autoregressive TTS by predicting all frames in parallel, trading model simplicity for a duration predictor and a length regulator.
- 07 Griffin-Lim and Classical Vocoders Griffin-Lim is an iterative phase-recovery algorithm that converts a magnitude spectrogram back into a waveform without any learned parameters, and understanding it clarifies exactly what neural vocoders had to replace and why.
- 08 HiFi-GAN and Neural Vocoders Neural vocoders convert acoustic feature representations into raw audio waveforms, and HiFi-GAN achieves near-human quality at 167x real-time speed using a multi-scale, multi-period GAN architecture.
- 09 Neural Audio Codecs Neural audio codecs compress waveforms into discrete token sequences using learned vector quantisation, enabling language models to generate speech token-by-token.
- 10 Prosody and Style Control How TTS systems encode and manipulate pitch, duration, energy, and speaking style so that synthesised speech sounds intended rather than merely intelligible.
- 11 Residual Vector Quantisation Residual vector quantisation stacks multiple codebooks to approximate a continuous audio embedding with increasingly fine-grained corrections, making it the compression backbone of modern neural audio codecs.
- 12 Streaming TTS and Latency Streaming TTS pipelines generate and deliver audio incrementally to cut time-to-first-audio from several seconds to under 300 ms, but doing so imposes hard trade-offs on chunk size, model architecture, and prosodic coherence.
- 13 Tacotron 2 Tacotron 2 is a two-stage neural TTS pipeline that converts text to mel spectrograms with a sequence-to-sequence model, then synthesises raw audio with a conditioned WaveNet vocoder, achieving near-human MOS scores.
- 14 Text Normalisation and Phonemisation Text normalisation converts raw written text into a speakable form, and phonemisation maps those words to phoneme sequences; together they determine what a TTS system says before any audio is generated.
- 15 The Acoustic Model and Vocoder Split Modern neural TTS splits the problem into two specialised sub-networks - an acoustic model that maps text to a compact spectral representation, and a vocoder that reconstructs a full audio waveform from that representation.
- 16 The Mel-Spectrogram Interface The mel spectrogram is the agreed-upon intermediate representation that decouples acoustic modelling from waveform generation in modern TTS pipelines.
- 17 WaveNet WaveNet is a fully autoregressive convolutional neural network that models raw audio waveforms one sample at a time, achieving near-human speech quality at the cost of extremely slow sequential generation.
- 18 Diffusion Models for Speech Diffusion models iteratively denoise random Gaussian noise into speech waveforms or mel-spectrograms, achieving sample quality that matches autoregressive vocoders at a fraction of the sequential compute cost.
- 19 VALL-E: TTS as Token Language Modelling VALL-E reformulates text-to-speech as a conditional language modelling problem over discrete audio codec tokens, enabling zero-shot voice cloning from a three-second recording by treating acoustic context the same way GPT treats a few-shot text prompt.
- 20 Zero-Shot Voice Cloning Zero-shot voice cloning synthesises speech in the voice of an unseen speaker from a short reference recording, without any fine-tuning at inference time.
Robotics & Embodied AI Vision-language-action models, action tokenisation, diffusion policies and sim-to-real. 5 concepts · 32 cards
- 01 Action Tokenisation and Representation How continuous robot actions become discrete tokens an autoregressive transformer can emit, from per-dimension binning to frequency-space compression.
- 02 Imitation Learning and Diffusion Policies Why cloning a demonstrator's actions drifts into unseen states, and how generative action models such as diffusion policies and action chunking control the drift.
- 03 Sim-to-Real Transfer Why robot policies are trained in simulation, why they break on real hardware, and how domain randomisation closes the reality gap by making the real world look like one more random draw.
- 04 RT-2 and Web-Scale Robot Learning RT-2 co-trains one transformer on internet vision-language data and robot trajectories by encoding actions as text tokens, transferring semantic web knowledge into robotic control.
- 05 Vision-Language-Action Models Turning a pretrained vision-language model into a robot policy that maps camera images plus a language instruction to motor actions, so the robot inherits web-scale semantic knowledge it could never learn from robot data alone.
AI for Science AlphaFold, protein language models, materials discovery, and the pitfalls of ML-for-science. 9 concepts · 52 cards
- 01 Machine Learning for Materials Discovery How graph neural networks screen millions of candidate crystals for stability and how machine-learning interatomic potentials approximate DFT cheaply enough to simulate them, plus why an in-silico "stable" material is not yet a real one.
- 02 Pitfalls in ML-for-Science The failure modes, chiefly data leakage, that make machine-learning results in scientific papers look stronger than they replicate, and the reporting standards proposed to catch them.
- 03 Protein Language Models How masked-language-model pretraining over amino-acid sequences produces structure and function signal, and how ESMFold trades some accuracy for dropping the MSA search that AlphaFold2 depends on.
- 04 AI for Formal Mathematics How proof assistants turn mathematics into a verifiable reward signal, what AlphaGeometry and AlphaProof achieved at the IMO, and why autoformalisation remains the bottleneck.
- 05 AlphaFold2 and the Protein-Folding Problem How a deep-learning system read co-evolution signal out of aligned protein sequences to predict 3D structure at near-experimental accuracy, and what it still cannot do.
- 06 AlphaFold3 and Biomolecular Co-Folding How AlphaFold3 dropped the protein-only structure module for a diffusion head that denoises raw atoms, letting one model co-fold proteins with ligands, nucleic acids, ions, and modified residues, and how the open reimplementations caught up.
- 07 Machine-Learned Interatomic Potentials How equivariant graph networks reach near-quantum accuracy at a fraction of the cost, why symmetry is designed in rather than learned, and what breaks when a potential leaves its training chemistry.
- 08 Neural Operators and PDE Surrogates Why learning a mapping between function spaces is different from fitting a network to a grid, how the Fourier neural operator achieves resolution invariance, and what a surrogate cannot promise.
- 09 Neural Weather Prediction How graph and transformer models trained on reanalysis data overtook physics-based forecasting on most verification targets, what they still depend on, and where the learned approach genuinely fails.