Early-Fusion Mixed-Modal Models

Almost every open VLM is a late-fusion model: a frozen vision encoder produces features, a small projector maps them into the language model's embedding space, and the language model, largely unchanged, treats them as prefix tokens. It works, it is cheap, and it inherits both models' pretraining. It also permanently separates the two modalities. The image is a foreign object translated into the text model's language, and the model can never generate an image because it has no way to speak back.

Early fusion removes the translator. Images are quantised into discrete tokens by a learned visual tokeniser, those tokens share a vocabulary with text, and a single transformer is trained from scratch on interleaved sequences with one next-token objective. Chameleon does exactly this, and reports state-of-the-art image captioning, text-only performance beating Llama-2 and competitive with Mixtral 8x7B, plus non-trivial image generation, all from one set of weights (Chameleon Team, 2024, arXiv:2405.09818).

What changes when the fusion moves earlier

In a late-fusion model, the image is summarised before the language model sees it. Whatever the encoder discarded is gone; the LLM cannot go back and look again with a different question in mind. In an early-fusion model, image tokens sit in the same residual stream as text tokens from layer 0, are attended to by every head, and get updated by every block. Attention between a question token and an image token is computed with weights trained on that exact interaction rather than on a projector fitted after the fact.

The generation direction is the real prize. Because image tokens are ordinary vocabulary items, the model can emit them, so text-to-image, image-to-image, and interleaved documents are all the same decoding problem. Any-to-any is a property of the token space, not a separate head.

The stability tax

Training one transformer on two modalities with very different token statistics diverges in ways pure text training does not. Image tokens from a codebook have a flatter, higher-entropy distribution than text; text tokens have long-tailed frequency and strong local predictability. The modalities compete for the same norm budget, and logits drift apart until softmax saturates.

Chameleon's fixes are architectural rather than heuristic: query-key normalisation to bound attention logit growth, and a reordering of the layer norms inside the block, both aimed at the divergence that appeared only in the mixed-modal setting. Reporting these as first-class contributions is itself informative; naive early fusion at scale does not train.

There is a middle path. Transfusion keeps one transformer but uses two objectives, next-token prediction on text and a diffusion loss on continuous image patches, avoiding quantisation entirely; at 7B parameters over 2T mixed-modal tokens it scales better than quantising images and running a language model over discrete codes (Zhou et al., 2024, arXiv:2408.11039).

When it breaks

The tokeniser is a hard ceiling. In a discrete early-fusion model, no amount of transformer capacity recovers detail the visual tokeniser threw away. Reconstruction error of the tokeniser sets a floor on generation quality and a ceiling on fine-grained perception, particularly for text rendered inside images and for faces.

Text ability regresses under a fixed budget. Every image token in the pretraining mix is a text token not seen. Early-fusion models are usually compared against text-only models trained on fewer tokens, and the honest question, whether mixed-modal pretraining costs text ability at matched compute, is rarely answered cleanly.

Cost of entry. Late fusion fine-tunes a projector on a million samples in a day on one node. Early fusion is a pretraining run. This, not any quality argument, is why the open ecosystem is overwhelmingly late fusion and the frontier labs are increasingly not.