Your RAG assistant gives confident answers that are subtly wrong. Where do you look first?

What the interviewer is testing

Whether you separate retrieval failure from generation failure, which is the diagnostic discipline that makes RAG debuggable.

The first thing to establish

Was the correct passage retrieved?

The two failures look identical to a user and require opposite responses:

• Not retrieved — no prompt engineering, no larger model and no better instructions will help. The fix is chunking, embedding, hybrid search or reranking.
• Retrieved and still wrong — retrieval is fine; the problem is the prompt, the model, or how the context was ordered and presented.

Answer this by logging the retrieved context alongside every response. If that logging does not exist, it is the first thing to build — a bad answer cannot be diagnosed without knowing what the model was shown.

If retrieval is the problem

Chunking that cut a table from its header or a clause from its condition. Missing lexical search — dense vectors miss exact identifiers, so a query for a policy reference or error code retrieves general passages. No reranking, so the right passage was in the candidate set but not in the top five. No query rewriting, so a follow-up question with a pronoun embedded to nothing useful.

If generation is the problem

Too much context — models attend less reliably to material in the middle of a long context, so fifty passages produce worse answers than five. Conflicting sources in the context with no recency or authority signal. No instruction to abstain, so the model answers from parametric knowledge when the context is insufficient. No citation requirement, which both degrades faithfulness and removes the user's ability to verify.

The measurement to put in place

A labelled set of real questions with known correct sources, measuring recall at k for retrieval separately from answer quality. Without it, changes are argued rather than evaluated — and roughly half the changes teams are confident about turn out to make things worse.

What a strong answer adds

Requiring citations in the output and validating that cited claims are supported by the retrieved context. That turns an unverifiable answer into a checkable one and gives users a way to catch the remaining errors themselves.

Common weak answers

Switching to a larger model. Adding more instructions to the prompt without establishing which half is failing.