Vision & Multimodal
ViT, CLIP, diffusion, SAM, and the vision-language models that read images as tokens.
12concepts
131flashcards
101minutes of reading
- 01 Contrastive Vision-Language: CLIP How a 400M image-text contrastive objective produced a shared embedding space that does zero-shot classification, retrieval, and grounding without any task-specific labels.
- 02 Multimodal Hallucination and Benchmark Validity Why vision-language models describe objects that are not in the image, and why a large share of multimodal benchmark scores can be reproduced without showing the model any image at all.
- 03 Native-Resolution Vision Encoding Why squashing every image to a fixed square destroyed text legibility in early VLMs, and how tiling, patch packing and dynamic resolution replaced it at the cost of an unbounded visual token budget.
- 04 Segment Anything (SAM) and Dense Prediction How promptable segmentation became a foundation-model task, what SAM's encoder-decoder split was designed for, and where it still loses to specialist models.
- 05 Video, Audio, and Any-to-Any Models How Whisper, V-JEPA, Sora-class video generators, MusicGen, and unified any-to-any models extend the multimodal stack beyond static images.
- 06 Vision Transformers (ViT) How treating an image as a sequence of patches let pure transformers beat CNNs once data crossed the 300M-image mark, and what the architecture gave up to get there.
- 07 Visual Document Retrieval Why the OCR-parse-chunk-embed pipeline loses exactly the documents that matter, and how late-interaction retrieval over page images replaces five brittle stages with one model.
- 08 Diffusion Models How learning to invert a noise process became the dominant generative recipe for images, video, and audio, and why Flow Matching and DiTs are reshaping the recipe in 2024.
- 09 Discrete Visual Tokenisers How VQ-VAE, VQGAN and FSQ turn an image into a short sequence of integers, why codebook collapse wrecks half of them, and what the compression ratio costs in reconstruction fidelity.
- 10 Early-Fusion Mixed-Modal Models The difference between bolting a vision encoder onto a frozen LLM and training one transformer over interleaved image and text tokens from scratch, and why the second approach destabilises training.
- 11 Flow Matching and Rectified Flow Why regressing a velocity field along straight noise-to-data paths replaced score matching in frontier image models, and what the straight-line claim does and does not guarantee.
- 12 Multimodal LLMs: LLaVA, Flamingo, GPT-4V The vision-encoder-plus-projector-plus-LLM recipe that dominates open multimodal models, why Flamingo's perceiver design still matters for video, and what native-multimodal frontier models do differently.