Chunk Boundary Strategy
How source documents are split for embedding, which determines whether retrieved passages are self-contained and coherent.
Chunking is the least glamorous part of a retrieval system and among the most consequential. A chunk that splits a sentence, a table or a procedure mid-way retrieves as noise.
Fixed-size chunking with overlap is the common default and the crudest: it ignores document structure entirely, and the overlap is a blunt hedge against splitting something important.
Better approaches, roughly in order of effort: structural chunking, splitting on headings, sections, list items and table boundaries, which respects how the document was written; semantic chunking, using embedding similarity between sentences to find topic shifts; and document-type-specific handling, since a legal contract, an API reference and a support transcript have entirely different natural units.
Two techniques that consistently help:
Contextual enrichment — prefixing each chunk with its document title and section path, so an isolated passage carries the context needed to interpret it. Cheap and effective.
Small-to-big retrieval — embed small precise chunks for matching, but return the larger surrounding section for generation. This separates the retrieval unit from the generation unit, which are optimised for different things.
The sizing trade: small chunks give precise matching and lose context; large chunks give context and dilute the embedding so matching degrades. There is no universal answer, which is why it must be measured against an evaluation set rather than chosen from a blog post.