Multimodal Hallucination and Benchmark Validity
Why vision-language models describe objects that are not in the image, and why a large share of multimodal benchmark scores can be reproduced without showing the model any image at all.
Show a vision-language model a photo of a kitchen counter with a bowl and a cutting board, and ask whether there is a knife. Many models say yes. Knives co-occur with cutting boards in the captioning corpora these models were trained on, and the language prior is a stronger signal than the pixels. This is object hallucination, and it is the cleanest demonstration that a VLM's answer is not necessarily a function of its image input.
Measuring it without letting the model wriggle
Free-form captions are hard to score, and early hallucination metrics were confounded by instruction phrasing and generation length. POPE reframes the measurement as polling: ask a series of balanced yes/no questions of the form "Is there a <object> in the image?", half about objects that are present and half about objects that are not, then report accuracy, precision, recall and F1 like any binary classifier. It is stable under prompt changes because the answer space is two tokens (Li et al., EMNLP 2023, arXiv:2305.10355).
The diagnostic value is in which negatives fail. POPE samples negative objects three ways: at random, by overall frequency in the dataset, and by co-occurrence with objects actually present. Models do best on random negatives and worst on co-occurring ones, and that ordering is the fingerprint of a language prior overriding perception. A model guessing from dataset statistics is nearly perfect on "is there a giraffe" in a kitchen and unreliable on "is there a knife".
The benchmark problem is worse than the hallucination problem
If a model can answer without the image, the benchmark measures something other than vision. This turns out to be common. GeminiPro scores 42.9% on MMMU with no visual input at all, and across six multimodal benchmarks LLMs without images beat the random baseline by more than 24 points on average; Sphinx-X-MoE reaches 43.6% on image-free MMMU, exceeding its own language backbone by 17.9 points, which points at training data leakage rather than world knowledge (Chen et al., 2024, arXiv:2403.20330).
MMMU itself is a serious benchmark: 11.5K college-level questions across six disciplines, 30 subjects and 183 subfields, with 30 image types, and at release GPT-4V and Gemini Ultra reached only 56% and 59% (Yue et al., CVPR 2024, arXiv:2311.16502). The follow-up is the more useful instrument. MMMU-Pro filters out questions text-only models can answer, expands the candidate options to blunt guessing, and adds a vision-only setting where the question itself is embedded in the image. Scores drop by 16.8 to 26.9 points across models (Yue et al., 2024, arXiv:2409.02813).
A 20-point drop from removing text shortcuts is the number to carry around when reading any multimodal leaderboard.
What to do about it
Run the blind baseline. Before trusting any multimodal eval, internal or public, score the same questions with the image removed. Whatever that number is, it is the floor your reported score should be discounted against. This costs one extra eval run and routinely changes conclusions.
Balance the negatives by co-occurrence. A hallucination suite whose negative objects are drawn at random will report a flattering number. Draw them from objects that frequently co-occur with what is actually present.
Separate perception from reasoning failures. When a model gets a chart question wrong, determine whether it misread a value or reasoned badly from a correctly read value. The fixes are unrelated: the first is a resolution and encoder problem, the second is a language-model problem.
Treat decoding as part of the system. Hallucination rates move with sampling temperature, beam settings and caption length, so a comparison between two models at different decoding settings is not a comparison of the models. Longer generations hallucinate more, mechanically, because each additional sentence is another opportunity to assert something unverified.
12 flashcards for this concept
Click a card to reveal the answer.