Vision & Multimodal
ViT, CLIP, diffusion, SAM, and the vision-language models that read images as tokens.
6concepts
32flashcards
51minutes of reading
- 01 Contrastive Vision-Language: CLIP How a 400M image-text contrastive objective produced a shared embedding space that does zero-shot classification, retrieval, and grounding without any task-specific labels.
- 02 Segment Anything (SAM) and Dense Prediction How promptable segmentation became a foundation-model task, what SAM's encoder-decoder split was designed for, and where it still loses to specialist models.
- 03 Video, Audio, and Any-to-Any Models How Whisper, V-JEPA, Sora-class video generators, MusicGen, and unified any-to-any models extend the multimodal stack beyond static images.
- 04 Vision Transformers (ViT) How treating an image as a sequence of patches let pure transformers beat CNNs once data crossed the 300M-image mark, and what the architecture gave up to get there.