You are building retrieval over an organisation's documents. Which three decisions most affect quality?

What is being tested

Whether you know which knobs actually move retrieval quality, rather than assuming the model choice dominates.

The three

1. Chunking. The parameter with the largest effect, and the one most often chosen arbitrarily.

Small chunks retrieve precisely and lose context; large chunks preserve context and dilute the signal. Overlapping chunks and preserving structural context — the section heading, the document title, prepended to each chunk — both help materially.

Chunking on semantic boundaries (sections, paragraphs) beats fixed character counts, which routinely split a sentence in half.

2. Hybrid retrieval with re-ranking. Pure vector search misses exact matches — a product code, an error identifier, a person's name — which are frequently exactly what someone is searching for.

Combining keyword and vector retrieval, then re-ranking the combined candidate set, consistently outperforms either alone. Omitting the keyword half is a common and avoidable weakness.

3. An evaluation set. Without a set of questions with known-good answers, quality is assessed by impression, and every change is a guess. This is the difference between tuning and hoping, and it is the step teams skip.

The decisions that matter but rank lower

Model choice, which matters less than chunking and carries a real switching cost: vectors from different models are not comparable, so changing the model means re-embedding the entire corpus. Store the source content and the model version alongside the vectors so re-embedding is possible.

Storage. At modest scale, a vector extension to an existing database beats a dedicated store, because the operational cost of an additional store is real and the threshold for needing a specialised one is higher than it appears.

Freshness. Embeddings must be regenerated when content changes — an indexing pipeline with lag that must be monitored like any projection.

The requirement that is not about quality at all

Access control at retrieval time. Retrieval must respect the requesting user's permissions, or the system becomes a data exfiltration tool with a natural language interface: a user asks a question and receives content they are not entitled to, entirely legitimately from the system's point of view.