Contextual Retrieval and Chunk Augmentation

Here is a chunk from a real filing: "The company's revenue grew by 3% over the previous quarter." Which company? Which quarter? The chunk cannot say, because the answer lived in a heading four pages earlier that the splitter discarded. Embed that sentence and it lands in a region of vector space shared by every revenue sentence ever written, retrievable by nothing in particular. This is context destruction, and it is the failure mode that survives after you have tuned chunk size, overlap, and splitter (see chunking strategies for retrieval).

Anthropic's contextual retrieval attacks it directly: before embedding, use a cheap model to write 50 to 100 tokens explaining where this chunk sits in its document, and prepend that to the chunk (Anthropic, 2024, Contextual Retrieval in AI Systems). The example above becomes "This chunk is from an SEC filing on ACME Corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter." Now it is retrievable by the query that was always meant to find it.

The mechanism, and where the cost went

The augmentation prompt takes the whole document plus one chunk and asks for a short situating statement. Naively that means re-reading the entire document once per chunk: a 200-page document split into 800 chunks would be read 800 times. Prompt caching is what makes this tractable, because the document is a fixed prefix and only the chunk varies, so the document's KV cache is computed once and read back at roughly a tenth of the price for every subsequent chunk (see prompt caching infrastructure). Anthropic put the resulting one-off indexing cost on the order of a dollar per million document tokens. This is a technique that existed conceptually for years and became practical when the cost structure changed.

The augmented text is used for both indices. Contextualised chunks are embedded for semantic search, and the same contextualised text is fed to BM25 for lexical search, because the situating sentence adds exactly the proper nouns, dates, and identifiers that lexical matching keys on.

What it actually buys

Source: Anthropic, 2024. These are the vendor's own measurements across their evaluation corpora; treat the ordering as robust and the absolute numbers as specific to their data.

The ordering is the useful part. Hybrid lexical-plus-semantic retrieval (see hybrid retrieval BM25 vector) and reranking (see reranking and cross-encoders) are not alternatives to contextualisation; they compose with it, and each stage keeps most of its independent gain.

When it breaks

• You pay for a document you may never query. Contextualisation is an indexing-time cost applied to everything ingested, regardless of what gets asked. For corpora where a thin slice of documents absorbs all traffic, per-query context injection at generation time may be cheaper.
• Documents longer than the augmenter's window need chunked context. For a 5-million-token corpus in a single logical document, "the whole document" cannot be a cache prefix; you fall back to section-level context, which recovers less.
• The augmenter can hallucinate the situating statement. A small model asked to place a chunk will confidently assign the wrong quarter or the wrong subsidiary, and that error is now baked into the index permanently and invisibly. Sampling and auditing a few hundred augmentations is cheap insurance.
• Re-indexing is required when the augmentation prompt changes. The prompt becomes part of your index schema. Treat it as versioned, migrated infrastructure, not a string in a notebook.
• It does not fix a bad chunk boundary. If a table has been split across two chunks so that headers and rows are separated, adding a sentence of context to each half does not reassemble the table. Contextualisation repairs missing situation, not broken structure.