Vision & Multimodal
ViT, CLIP, diffusion, SAM, and the vision-language models that read images as tokens.
12concepts
131flashcards
101minutes of reading
- 01 Contrastive Vision-Language: CLIP How a 400M image-text contrastive objective produced a shared embedding space that does zero-shot classification, retrieval, and grounding without any task-specific labels.
- 02 Multimodal Hallucination and Benchmark Validity Why vision-language models describe objects that are not in the image, and why a large share of multimodal benchmark scores can be reproduced without showing the model any image at all.
- 03 Native-Resolution Vision Encoding Why squashing every image to a fixed square destroyed text legibility in early VLMs, and how tiling, patch packing and dynamic resolution replaced it at the cost of an unbounded visual token budget.
- 04 Segment Anything (SAM) and Dense Prediction How promptable segmentation became a foundation-model task, what SAM's encoder-decoder split was designed for, and where it still loses to specialist models.
- 05 Video, Audio, and Any-to-Any Models How Whisper, V-JEPA, Sora-class video generators, MusicGen, and unified any-to-any models extend the multimodal stack beyond static images.
- 06 Vision Transformers (ViT) How treating an image as a sequence of patches let pure transformers beat CNNs once data crossed the 300M-image mark, and what the architecture gave up to get there.
- 07 Visual Document Retrieval Why the OCR-parse-chunk-embed pipeline loses exactly the documents that matter, and how late-interaction retrieval over page images replaces five brittle stages with one model.