Vision & Multimodal
ViT, CLIP, diffusion, SAM, and the vision-language models that read images as tokens.
12concepts
131flashcards
101minutes of reading
- 01 Diffusion Models How learning to invert a noise process became the dominant generative recipe for images, video, and audio, and why Flow Matching and DiTs are reshaping the recipe in 2024.
- 02 Discrete Visual Tokenisers How VQ-VAE, VQGAN and FSQ turn an image into a short sequence of integers, why codebook collapse wrecks half of them, and what the compression ratio costs in reconstruction fidelity.
- 03 Early-Fusion Mixed-Modal Models The difference between bolting a vision encoder onto a frozen LLM and training one transformer over interleaved image and text tokens from scratch, and why the second approach destabilises training.
- 04 Flow Matching and Rectified Flow Why regressing a velocity field along straight noise-to-data paths replaced score matching in frontier image models, and what the straight-line claim does and does not guarantee.
- 05 Multimodal LLMs: LLaVA, Flamingo, GPT-4V The vision-encoder-plus-projector-plus-LLM recipe that dominates open multimodal models, why Flamingo's perceiver design still matters for video, and what native-multimodal frontier models do differently.