Multimodal LLMs: LLaVA, Flamingo, GPT-4V

A language model is a sequence-in, sequence-out function. The cleanest way to add vision is to turn images into more tokens, then let the language model do what it already does. That sentence describes 95% of open multimodal models shipped since 2023. The interesting design choices are: which vision encoder, how to project image features into the LLM's token space, whether to interleave images with text or keep them separate, and what training data and recipe turn this Frankenstein into something that actually follows instructions about images.

The dominant recipe

LLaVA (Liu et al, April 2023) crystallised the playbook:

1. Vision encoder. A frozen pretrained CLIP-style ViT (CLIP ViT-L/14 in the original LLaVA). Produces a grid of patch features (e.g. 24x24 = 576 tokens at 336x336 input).
2. Projector. A small MLP (originally one linear layer, later a two-layer MLP) maps the vision feature dim to the LLM's token embedding dim.
3. LLM. A pretrained instruction-tuned LLM (Vicuna in the original, Llama / Mistral / Qwen in later forks). Frozen at first, fine-tuned later.

At inference, prepend the projected image tokens to the text tokens and feed the lot into the LLM. The model treats image tokens as if they were text - same attention, same position encoding, same loss.

The recipe is so standard that adapting a new LLM to vision is now a long weekend's work, not a research project.

Why LLaVA shipped quickly

LLaVA's contribution was less the architecture (others had stitched encoders to LLMs before) and more the data. The team used GPT-4 (text-only) to generate visual instruction-tuning data: feed it image captions and bounding boxes from COCO, ask it to write plausible Q&A and reasoning chains, then train the visual model to produce those outputs given the actual image. 158k synthetic instruction-following examples, trained in a day on 8 A100s.

Two consequences:

• The training recipe is reproducible. Anyone with a small GPU budget and an LLM API can generate equivalent data. The open multimodal community has been iterating on this loop ever since.
• Instruction-following capability transferred almost for free. Because the LLM was already instruction-tuned and the projector is small, only the projector and a low-rank LLM adapter need to learn how to consume images. The world knowledge in the LLM is intact.

LLaVA's quality on visual question answering jumped through several version bumps (1.0 -> 1.5 -> Next -> OneVision) by improving the projector, raising input resolution, and adding more diverse instruction data. The architecture stayed the same.

Flamingo's perceiver resampler and interleaving

Flamingo (Alayrac et al, April 2022) predates LLaVA and made different choices that still matter:

• Frozen LLM and frozen vision encoder. Only inserted "gated cross-attention" layers in the LLM are trained. The base LLM never sees gradients.
• Perceiver resampler. A small transformer that compresses an arbitrary number of vision features (e.g. 256 features from a static image, 8 frames x 256 from video) into a fixed 64 visual tokens. This decouples the LLM's compute from input image count.
• Interleaved input. Native support for arbitrary sequences of [text, image, text, image, ...]. Few-shot in-context learning across images works the same way as in-context learning across text.

The interleaved-input design is what made Flamingo good at video (sample frames and interleave them with timestamps) and multi-image reasoning. Most modern frontier multimodal models (GPT-4o, Gemini, Claude 3.5) inherit interleaved support. LLaVA-style "one image up front, then text" remains common in open models because it is simpler to train.

The perceiver resampler is also re-emerging in newer open models (Idefics, BLIP-2's Q-Former is a close cousin) wherever input resolution is variable and you want to cap visual token count.

Native-multimodal frontier models

GPT-4V (September 2023) was the first frontier-grade multimodal chat model and was almost certainly a bolt-on - CLIP-style encoder plus the GPT-4 backbone. GPT-4o (May 2024) and Gemini (December 2023) onward are different: trained from scratch on interleaved text, image, audio, and (for Gemini and GPT-4o) speech tokens. The architectural details are not public, but the observable behaviours suggest:

• A single unified tokeniser or close approximation across modalities. The same model can take audio in and emit audio out in one pass (GPT-4o's voice mode), which strongly implies no separate ASR/TTS pipeline.
• Higher visual token budgets. Both Gemini and GPT-4o handle high-resolution images (e.g. screenshots of code) without obvious tiling artefacts. Open models like Qwen2-VL with dynamic resolution are catching up.
• Joint pretraining at scale. The behaviour on cross-modal tasks (describe what is in this audio while also describing the image) is qualitatively different from cascaded systems.

The trade-off for open models is brutal: native-multimodal pretraining requires datasets and compute well beyond what any open lab has been able to assemble. The LLaVA-style two-stage recipe (pretrain a strong unimodal model, then fuse) remains the only viable open path.

The multimodal evaluation problem

Benchmarking these models is genuinely hard. The standard suite:

• MMMU (Yue et al, 2023). 11.5k multimodal questions across 6 disciplines and 30 subfields, drawn from college exams. The headline benchmark; GPT-4V scored 56%, Gemini Ultra 59%, humans 89%. Saturation is still some way off but progress is fast - 2025 frontier models cross 70%.
• MM-Vet. Tests six capabilities (recognition, OCR, knowledge, reasoning, generation, spatial) with open-ended responses graded by GPT-4. Less saturated than VQA.
• MMBench and SEED-Bench. Large multiple-choice batteries; easier to grade, more prone to shortcut learning.
• RealWorldQA and BLINK. Tests of spatial reasoning and core visual understanding where current models still underperform humans by large margins.

Three failure modes recur:

1. Multiple-choice gaming. Models exploit answer-format biases without looking at the image.
2. Caption-only baselines. A surprising number of "visual" questions can be answered from a text caption alone. If a caption-only LLM solves the benchmark, the benchmark is not measuring vision.
3. English-only and Western-image bias. MMMU and most others are English; multilingual multimodal eval is sparse.

Where it falls down

• Spatial reasoning. Open multimodal models do badly on tasks that require precise spatial relations (left/right, in-front-of, counting) - the failure inherited from CLIP-style encoders.
• Long video and dense temporal reasoning. Compressing a 1-minute video into a perceiver-resampled token budget loses fine-grained temporal events. Frontier closed models do better; open models are 12-18 months behind.
• Tool use with vision. Models can describe a screenshot but struggle to use it as state in a multi-step agent loop. Computer-use models (Anthropic's Claude Computer Use, OpenAI Operator) are early attempts at this.

Further reading

• Flamingo: a Visual Language Model for Few-Shot Learning - Alayrac et al 2022, perceiver resampler and interleaved input.
• Visual Instruction Tuning - Liu et al 2023, the LLaVA paper that set the open recipe.
• haotian-liu/LLaVA - reference implementation and instruction-tuning data.
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark - Yue et al 2023, the dominant frontier benchmark.