A retrieval system over code and documentation performs poorly. Which chunking decisions matter, and what is the common mistake?

The common mistake

Fixed-size chunks with arbitrary boundaries. Splitting on a character or token count cuts through functions, tables, procedures and arguments, producing chunks that contain a conclusion without its conditions, a code block without its signature, or a step without its prerequisite.

The retrieved content is then technically relevant and semantically useless — and the resulting wrong answers look like model failures, which sends the team optimising the wrong layer.

The decisions that matter

1. Chunk on structure, not on size. Documents split at headings and sections; code split at function, class or file boundaries. The structure exists because it carries meaning, and respecting it preserves that meaning.

2. Attach parent context to every chunk. Document title, section path, file path, repository, and the surrounding declaration for code. A chunk retrieved without knowing where it came from cannot be interpreted by the model or verified by the user.

3. Overlap deliberately where boundaries are genuinely ambiguous, accepting some duplication in exchange for not losing cross-boundary meaning.

4. Vary chunk size by content type. A reference table, a prose explanation and a function have different natural units. One size across a heterogeneous corpus is wrong for most of it.

5. Retrieve small, provide large. Index fine-grained chunks for precise matching, then supply the model with the enclosing section or file. This decouples retrieval granularity from generation context and is frequently the single highest-return change.

For code specifically

• Preserve the signature with the body, or the retrieved code cannot be used.
• Keep imports and type definitions reachable, since a function without its types is ambiguous.
• Index the surrounding documentation and comments with the code, because the intent is usually in the prose and the mechanism in the code.
• Respect language structure using a parser rather than heuristics.

The evaluation requirement

Chunking cannot be tuned without measurement. A held-out set of real questions with known-correct source locations, and a metric for whether retrieval surfaced them, is what turns chunking from taste into engineering.

Teams without one adjust chunk sizes based on anecdote and cannot tell improvement from noise — which is the same reason evaluation is the most commonly deferred and most consequential omission in these systems.