Chunking Strategies for Retrieval

A retrieval system can only return things it indexed. If a 40-page contract is stored as one vector, every query about it returns the whole contract and the generator drowns. If the same contract is stored as 800-character fragments, the clause that says "the Term shall be five years" retrieves cleanly, but the fragment that says "such period may be extended" has lost the antecedent for such period and is now actively misleading. Chunking is the choice between those two failures, and it is made before a single embedding is computed.

The size and overlap dials

The default strategy is fixed-size chunking with overlap: split at N tokens, repeat the last K tokens at the head of the next chunk. Typical production values sit around 256 to 512 tokens with 10 to 20 percent overlap, and the reason is mechanical rather than principled. Embedding models compress a chunk into one vector; the longer the chunk, the more topics that vector must average over, and averaged vectors sit near the centroid of the corpus where they match everything weakly and nothing strongly. Short chunks give sharp vectors and lose context. Overlap buys back some of the context at the cost of duplicating tokens in the index and returning near-identical neighbours at query time.

Recursive character splitting improves on this without changing the economics: split on paragraph breaks first, then sentences, then words, descending only when a piece still exceeds the budget. The chunks land on natural boundaries most of the time, which is worth more than any tuning of N.

Structure beats statistics

The strongest gains come from respecting the document's own structure. Markdown headings, HTML sections, legal clause numbers, and code function boundaries are hand-authored semantic segmentation, already in the file, free to use. A chunker that splits a codebase on function definitions and prefixes each chunk with its file path and class name will beat a token-count splitter on the same corpus without any model involved.

Two refinements are worth knowing:

• Small-to-big (or parent-document) retrieval. Index small, precise chunks for matching, but return the larger parent section to the generator. Retrieval precision and generation context are different requirements, and this decouples them.
• Hierarchical summarisation. RAPTOR clusters chunks, summarises each cluster, embeds the summaries, and recurses, producing a tree where a query can match at whatever level of abstraction it was asked (Sarthi et al., 2024, RAPTOR, arXiv:2401.18059). It answers "what is this document about" questions that no leaf chunk can.

Restoring the context a chunk lost

Two methods attack the dangling-reference problem directly.

Contextual retrieval asks an LLM to write a short situating sentence for each chunk given the whole document, prepends it, then indexes the result. Anthropic reported that contextual embeddings plus contextual BM25 cut top-20 retrieval failures by 49 percent, and by 67 percent when a reranker was added on top (Anthropic, 2024, Contextual Retrieval). The cost is one LLM call per chunk at index time, which prompt caching makes affordable but not free.

Late chunking inverts the order of operations: run the whole document through a long-context embedding model, then pool token embeddings into chunk vectors after the transformer rather than before (Günther et al., 2024, Late Chunking, arXiv:2409.04701). Each chunk vector is computed with attention over the full document, so pronouns and abbreviations resolve. It needs no extra LLM calls, but requires an embedding model whose context window covers the document.

When it breaks

Chunking failures are silent. The pipeline returns something for every query, so a bad chunk boundary shows up as a subtly wrong answer, not an error. Three recurring traps:

• Tables and code split mid-structure. A chunk containing rows 40 to 60 of a table with no header is unusable, and the model will still summarise it confidently.
• Overlap masquerading as recall. Heavy overlap fills the top-k with three shifted copies of the same passage, so effective k collapses. Deduplicate by source span before reranking.
• Re-chunking is a re-index. Chunk size is not a runtime parameter. Changing it means re-embedding the corpus, which is why the decision deserves an evaluation set before it is frozen rather than after.

Chunk size interacts with everything downstream: with the index through corpus cardinality, with the reranker through what it has to work with, and with the generator through context rot. Tune it against end-to-end answer quality, never against retrieval metrics alone.