Native-Resolution Vision Encoding

Feed a 1920×1080 screenshot of a spreadsheet to LLaVA-1.5 and the model resizes it to 336×336 before the vision encoder ever sees it. A cell that occupied 11 pixels of height now occupies 3. No amount of language-model capacity recovers characters that were destroyed by a bilinear resize. This one preprocessing step, inherited unexamined from ImageNet-era classification, was the dominant failure mode of open vision-language models through 2023.

The fix looks trivial and is not: let the encoder see the image at something closer to its real resolution. The reason it is hard is that a Vision Transformer's sequence length is set by the image, and the language model pays for every one of those tokens.

The token arithmetic

A ViT cuts an \(H \times W\) image into non-overlapping \(p \times p\) patches, so the visual sequence length is

CLIP ViT-L/14 at 336 pixels gives \(24 \times 24 = 576\) tokens, which is what LLaVA-1.5 prepends to the prompt (Liu et al., 2023, arXiv:2310.03744). Double the resolution and \(N\) quadruples. Since the language model's prefill attention is quadratic in total sequence length, a 4× resolution increase costs roughly 16× more attention FLOPs in the decoder before a single output token is emitted.

That is the entire tension. Resolution buys legibility; tokens cost money.

Three ways to spend the budget

Tiling (AnyRes). Cut the image into fixed-size crops, encode each independently with the unchanged pretrained encoder, and prepend a downsampled thumbnail so the model retains global layout. LLaVA-NeXT selects from a grid set of {2×2, 1×{2,3,4}, {2,3,4}×1}, supporting inputs up to 672×672, 336×1344 and 1344×336 (LLaVA-NeXT, January 2024). InternVL 1.5 pushes the same idea to 1–12 tiles of 448×448 in training and up to 40 tiles at test time, which covers 4K input, and applies a pixel-shuffle that folds each 448×448 tile down to 256 tokens instead of 1,024 (Chen et al., 2024, arXiv:2404.16821).

Patch packing (NaViT). Rather than crop, keep the native aspect ratio and pack variable-length patch sequences from several images into one training sequence, using a block-diagonal attention mask so images cannot attend across each other. This is the same sequence-packing trick used in LLM pretraining, applied to vision. NaViT reports improved training efficiency for large-scale supervised and contrastive pretraining, plus the ability to trade test-time cost against accuracy on a smooth curve (Dehghani et al., 2023, arXiv:2307.06304).

Native dynamic resolution. Train the encoder from the start on variable resolutions with interpolatable or factorised position embeddings, then simply emit however many tokens the image implies. Qwen2-VL merges each 2×2 group of 14-pixel patches into one visual token, so one token covers a 28×28 pixel block, and pairs this with M-RoPE, which splits rotary position indices into temporal, height and width components so images and video share one positional scheme (Wang et al., 2024, arXiv:2409.12191). Pixtral 12B trains a vision encoder from scratch specifically to ingest images at natural resolution and aspect ratio (Agrawal et al., 2024, arXiv:2410.07073).

Where it breaks

Tile seams cut through content. A crop boundary that falls mid-word or mid-table-row splits evidence across two independently encoded tiles with no cross-tile attention inside the encoder. The language model has to stitch them, and it often stitches wrongly on long numbers and multi-column layouts. The thumbnail helps with layout but carries none of the fine detail.

Position embeddings do not extrapolate for free. A ViT pretrained at 224 has learned absolute position embeddings for a 16×16 grid. Interpolating them to a 64×64 grid is a distribution shift; it works, but only after fine-tuning at the target resolution, which is why "just increase the resolution" is a retraining project rather than an inference flag.

Cost becomes adversarial. With native resolution, a user controls your bill by choosing an image size. An unbounded token count per image means a single 4K page can consume more context than the entire conversation, so production systems cap tiles, and that cap silently reintroduces the downsampling problem for exactly the documents that needed the resolution.

More tokens stop paying off quickly. Visual tokens receive very little attention mass in the deeper layers of the language model; FastV exploits this by pruning them after early layers and reports large FLOP reductions with modest quality loss (Chen et al., 2024, arXiv:2403.06764). If half the visual tokens can be discarded at layer 2 with little damage, most of what the extra resolution bought was already redundant by the time the model started reasoning.