A retrieval-augmented system over enterprise documents gives confidently wrong answers. Which failure modes are responsible, and where should engineering effort go?

The failure modes, in order of likelihood

1. Retrieval did not surface the right content. By far the most common cause. The answer exists and the retrieval step did not find it, so the model answered from its own priors. Symptoms: plausible answers unrelated to the corpus.

2. Retrieved content is outdated. The index contains a superseded document and the model summarises it faithfully. In an enterprise corpus, superseded documents typically outnumber current ones, and nothing about the text distinguishes them.

3. Chunking destroyed the context. A chunk containing a conclusion without its conditions produces an answer that is technically present in the corpus and wrong in context. This is the failure that most looks like a model problem and is not.

4. Conflicting sources. Several documents disagree, retrieval returns some of them, and the model reconciles them silently rather than surfacing the conflict.

5. The answer genuinely is not in the corpus, and the system had no mechanism to say so.

Where effort should go

Retrieval quality, overwhelmingly. The output quality is dominated by what reaches the model. Practically:

• Hybrid retrieval — semantic and keyword — because pure semantic search misses exact identifiers, error codes, product names and acronyms, which are exactly what enterprise questions contain.
• Reranking the candidate set, which is usually the highest-return single addition.
• Chunking that preserves structure — respecting headings and sections, with parent context attached — rather than fixed-size splits.
• Freshness and authority as ranking signals, plus explicit archiving of superseded documents. In an enterprise corpus this frequently matters more than semantic relevance.
• Metadata filters — space, team, document type, date — so retrieval is scoped before it is ranked.

The mechanisms that address confident wrongness directly

• Citations to the retrieved chunks, so a user can verify. This converts an unverifiable assertion into a checkable one and is the highest-value product affordance.
• An explicit abstention path when retrieval confidence is low. "I could not find this" is a far better answer than a fluent invention, and it must be designed rather than hoped for.
• Surfacing conflict rather than silently reconciling it.

The non-negotiable

Permission filtering against current truth at retrieval time. An index built before a permission change, or filtering applied after ranking, leaks content between users. This is a security property, not a quality one, and it must fail closed.