Late Interaction and Multi-Vector Retrieval

Single-vector retrieval and cross-encoder reranking sit at opposite ends of a tradeoff. The bi-encoder compresses a passage into one vector, which is cheap and lossy. The cross-encoder reads query and passage together, which is accurate and cannot be precomputed. Late interaction is the architecture that refuses the choice: keep a vector per token, precompute all of them, and defer only the cheap comparison to query time.

MaxSim

ColBERT encodes a query into \(|q|\) token vectors and a document into \(|d|\) token vectors, then scores the pair by, for each query token, taking its maximum similarity against any document token and summing (Khattab & Zaharia, SIGIR 2020, arXiv:2004.12832):

Every document vector is computed offline, exactly as in a bi-encoder. The only query-time work is a matrix product and a max reduction, no transformer over the pair. What survives is fine-grained term matching: a query token for "arbitration" finds the document token for "arbitration" directly instead of hoping it survived mean pooling into a single 1024-dimensional average. Late interaction sits much closer to cross-encoder quality than to bi-encoder quality on BEIR-style out-of-domain benchmarks, which is the property that made it interesting.

The storage problem, and how it got fixed

The obvious objection is size. A 300-token passage becomes 300 vectors instead of one. At 128 dimensions in float32 that is roughly 150 KB per passage against 0.5 KB for a single 128-dim vector, and the naive index is two orders of magnitude larger.

ColBERTv2 attacked this with residual compression: cluster the token vectors, store a centroid id plus a 1 or 2 bit quantised residual per dimension, cutting the index by roughly 6 to 10 times with little quality loss (Santhanam et al., NAACL 2022, arXiv:2112.01488). PLAID then made retrieval fast by pruning candidate documents using centroid information before touching any residuals, cutting latency by several times on CPU and GPU (Santhanam et al., CIKM 2022, arXiv:2205.09707). Mainstream vector stores have since added native multi-vector support, so late interaction no longer requires a bespoke engine.

ColPali and the document-as-image turn

The most consequential recent use of late interaction is not text at all. ColPali feeds page images to a vision-language model, keeps the patch embeddings as the multi-vector representation, and applies ColBERT-style MaxSim against query token vectors (Faysse et al., 2024, arXiv:2407.01449). This deletes the entire PDF parsing pipeline, OCR, layout detection, table extraction, reading-order reconstruction, and replaces it with a screenshot. On the ViDoRe benchmark the authors introduced, it beats text-extraction pipelines on visually rich documents, where the information lives in a chart or a table's geometry that OCR flattens into noise.

For a corpus of scanned reports, slide decks and financial filings, this is a structural change to how the index is built, not a tuning improvement.

Where it fits

A common production shape is bi-encoder or hybrid retrieval for the first stage, late interaction as a cheap high-quality second stage over a few hundred candidates, and a cross-encoder only if the last few points of precision justify it.

When it breaks

• Index size is still the binding constraint. Compressed, a multi-vector index typically lands several times larger than a single-vector one. On a billion-passage corpus that decides the architecture by itself.
• Operational maturity is lower. Filtering, hybrid fusion, incremental updates and multi-tenancy are better supported for single-vector indexes in most engines.
• Query-token cost. Score computation scales with query length, so long queries and long documents multiply.
• It does not fix retrieval you never attempted. Late interaction improves ranking within candidates; it does not compensate for chunking that split the answer in half.