Visual Document Retrieval

The standard enterprise RAG pipeline over PDFs has five stages before a single embedding exists: PDF parsing, layout detection, OCR, optional table and figure captioning, then chunking. Each stage has its own error rate and the errors compound multiplicatively. Worse, the stages are lossy in a structured way. A bar chart becomes either nothing or a one-line caption; a merged-cell financial table becomes a scrambled text run; a scanned form loses the association between label and field. The documents that most need retrieval, the visually dense ones, are exactly the ones the pipeline mangles.

Retrieve on the pixels

ColPali skips the pipeline. A page is rendered as an image, passed through a vision-language model, and the resulting patch embeddings are stored directly as the document representation. PaliGemma-3B produces 1,024 patches per page, each projected to 128 dimensions (Faysse et al., ICLR 2025, arXiv:2407.01449).

Scoring is ColBERT-style late interaction rather than a single dot product. For query token embeddings \(\{q_i\}\) and page patch embeddings \(\{d_j\}\):

Each query token finds the patch it matches best, and those maxima are summed. This is what makes the approach work on documents: the token "EBITDA" can align to the one patch containing that cell without the page's other 1,023 patches diluting it, which is precisely what a single pooled page vector cannot do.

The reported numbers are the argument. On the ViDoRe benchmark ColPali reaches 81.3 nDCG@5 on average, against 67.0 for an Unstructured pipeline with captioning plus BGE-M3, with the gap widening on visually complex sets such as InfographicVQA (81.8 versus 71.9). Indexing runs at 0.39 seconds per page against 7.22 seconds for the parsing pipeline, and query encoding takes about 30 ms.

The cost is storage and it is not small

Multi-vector retrieval trades index size for accuracy. At 1,024 vectors of 128 dimensions in float16, a page costs about 257 KB, which the paper measures directly. A million-page corpus is therefore roughly 250 GB of index for a collection whose extracted text would fit in a few gigabytes.

Three mitigations are standard, and each gives something back. Binary or scalar quantisation of the vectors cuts storage by 8–32× with modest accuracy loss. Pooling patches, by row or by cluster, reduces the vector count per page. Two-stage retrieval uses a cheap single-vector or BM25 first pass and applies late interaction only to a few hundred candidates, which keeps the MaxSim cost bounded regardless of corpus size.

When it breaks

Long documents still need page-level granularity. The unit of retrieval is a page image, so an answer spanning a table that runs across a page break is split, and there is no chunk-overlap trick that fixes it cleanly.

Small text falls below the patch grid. The resolution the VLM encodes at determines the smallest legible glyph. Dense legal footnotes and 6-point table footers can be genuinely unreadable at the encoder's native grid, and raising resolution multiplies both index size and encoding cost.

Generation still needs the content. Retrieval returns a page image; answering a question about it requires a VLM strong enough to read it, so the pipeline's quality ceiling moves from the parser to the generator. Text-only LLMs cannot consume the retrieved unit at all.

Metadata and filters get harder. Structured filtering by date, author or section normally rides on the parsed text. Discard the parse and you either keep a parallel lightweight extraction or lose faceted search, which is a common late surprise in production.