Multimodal Prompting

A page of text costs perhaps 700 tokens. A screenshot of the same page can cost several thousand, and how many depends on a resolution policy you may not control. Prompting a vision-language model is prompting with a second modality whose cost, ordering, and failure modes have almost nothing in common with text.

Images are tokens, and resolution is a budget decision

Vision encoders turn an image into a sequence of patch embeddings that occupy positions in the same context window as text. Older models resized every image to a fixed resolution, which wasted tokens on small images and destroyed detail in large ones. Current models process native resolutions and emit a variable number of visual tokens, so a dense screenshot legitimately costs more than a photograph (Wang et al., 2024, Qwen2-VL, arXiv:2409.12191).

The practical consequences are direct. Downscaling an image below the threshold where text strokes survive turns an OCR task into guesswork, and no prompt fixes it. Cropping to the region of interest before sending costs nothing and improves both accuracy and price. For documents, a real OCR pass plus text is often cheaper and more accurate than asking the model to read pixels, and the right architecture is frequently both: send the crop and the extracted text.

Ordering, references, and grounding

Interleaving matters. Placing the question after the image generally works better than before it, because the model's answer is conditioned on everything to its left, and a question asked before the image cannot attend to detail it has not yet seen. With several images, unlabelled ones are hard to refer to; label them explicitly in the text ("Image 1 is the before state") and the model's references stop drifting.

For anything spatial, mark the image rather than describing the region in words. Overlaying numbered marks on segmented regions and then asking about "region 4" substantially improves grounding, because it converts a spatial reference problem into a symbolic one (Yang et al., 2023, Set-of-Mark Prompting, arXiv:2310.11441). This is the same trick that makes GUI agents workable: annotate the screenshot with element IDs and let the model name an ID instead of predicting coordinates.

Object hallucination is the characteristic failure

Vision-language models describe objects that are not in the image, and they do so systematically rather than randomly: the hallucinated objects are the ones that are frequent in the instruction-tuning data or that commonly co-occur with objects that are present. A kitchen photograph acquires a refrigerator; a street scene acquires a traffic light. The POPE evaluation exposes this by polling the model with balanced yes/no questions about object presence, including objects chosen by co-occurrence statistics, which is far more discriminating than scoring generated captions (Li et al., 2023, Evaluating Object Hallucination in Large Vision-Language Models, arXiv:2305.10355).

The prompting counter-measures are modest but real: ask for grounded answers ("quote the text you can actually read"), allow an explicit "not visible in the image" option, and ask verification questions about specific objects instead of accepting a free-form description.

When it breaks

• Counting and fine spatial relations. Counting more than a handful of objects, reading a chart's exact values, and reasoning about relative position are consistently weak. Verify against a tool where the answer matters.
• Prompt injection through pixels. Text rendered inside an image is instruction-shaped input from an untrusted source. A screenshot containing "ignore previous instructions" is a live injection vector, and the usual text-side filters never see it.
• Cost surprises at scale. A per-image token count that varies with resolution makes capacity planning harder than for text. Measure the distribution of image sizes in real traffic before pricing.
• Chain of thought helps less than in text. Verbal reasoning steps do not re-examine the pixels; they elaborate on the first encoding. If the encoder missed the detail, extended reasoning produces a more articulate wrong answer.