A business unit wants an assistant answering questions from 200,000 internal documents. They ask whether to fine-tune a model or use retrieval. How do you decide?
Show the full answer Hide the answer
Retrieval, for this requirement, and the reasoning is not about quality
Four properties decide it:
Freshness. Documents change. Retrieval reflects a change as soon as the index updates; fine-tuning requires retraining, so the model is stale between runs.
Attribution. Users need to know which document an answer came from, and to follow it. Retrieval provides this structurally; a fine-tuned model cannot cite reliably and will produce plausible references that do not exist.
Access control. Different users may see different documents. Retrieval can filter by the user's permissions at query time. A fine-tuned model has absorbed everything it was trained on, and there is no way to unlearn a document for one user.
Cost of being wrong. Retrieval failures are visible — the wrong document was retrieved. Fine-tuned failures are confident and unattributable.
Fine-tuning addresses a different problem: style, format and domain vocabulary, not knowledge that must be current and attributable.
Where the actual work is
Not the model. The retrieval quality, and it is where these projects fail:
- Chunking. How documents are split determines what can be retrieved. A policy split mid-clause retrieves half an answer.
- Retrieval strategy. Semantic search alone misses exact terms — product codes, policy numbers. Hybrid with keyword search is usually materially better.
- Ranking, because the first few chunks are what the model sees.
- Evaluation. A test set of real questions with known correct sources, run on every change. Without it, quality is anecdote.
The properties to design in from the start
Access control at retrieval time, mirroring document permissions. Citation of every claim. A refusal path when nothing relevant is retrieved — a system that answers anyway is worse than one that says it does not know. And logging of question, retrieved chunks and answer, for evaluation and for investigating complaints.
What to tell them
The model is the commodity part. Budget for document preparation, retrieval evaluation and access control integration, which is where the effort and the risk sit.