An internal AI assistant gives confidently wrong answers. The team wants to upgrade to a better model. What do you check first?
Show the full answer Hide the answer
Establish whether the model ever saw the right content
Log the retrieved chunks alongside each answer, then take the wrong answers and check: was the correct source retrieved?
Two very different diagnoses follow, and upgrading the model only helps with one of them.
Correct source retrieved, wrong answer produced. A generation problem. A better model may help; so may a better prompt, and so may instructing it to answer only from the provided context and to refuse otherwise.
Correct source not retrieved. A retrieval problem, and no model upgrade fixes it. This is the more common diagnosis by a wide margin.
The retrieval faults, in order of frequency
Chunking. The answer spans a boundary, or a policy was split mid-clause so only half is retrievable. Chunk on document structure rather than on a fixed token count.
Semantic search alone. It misses exact terms — product codes, policy numbers, error identifiers — which is precisely what internal questions contain. Hybrid retrieval with keyword search is usually a large improvement.
Ranking. The right chunk is retrieved at position twenty and the model sees five.
Stale or missing content. The document was never indexed, or the index is behind.
Build the evaluation set before changing anything
Fifty real questions with their known correct sources. Measure retrieval accuracy — was the right chunk in the top k — separately from answer quality.
Without this, every change is anecdote, and a model upgrade will be judged on the three questions someone happened to retry.
The two properties that matter more than accuracy
Citation, so a user can check. A refusal path when nothing relevant is retrieved — a system that answers anyway is worse than one that says it does not know, because confident wrong answers are what destroys trust in the assistant.
What to tell the team
Spend the effort on retrieval and evaluation. The model is the commodity part, and upgrading it without an evaluation set means you will not be able to tell whether it helped.