A client wants an assistant that answers questions from 50,000 internal documents which change weekly. RAG or fine-tuning? What actually determines the quality?

What the interviewer is testing

Whether you understand what each technique actually does, and whether you know that RAG quality is a retrieval problem.

Why RAG

Three decisive reasons, only the first of which is usually given:

Freshness. The documents change weekly. Updating RAG is a re-index of the changed documents — minutes. Updating a fine-tune is a training run and an evaluation cycle — days, every week, forever.

Permissions. This is the one that settles it for enterprise content. Internal documents have access controls, and RAG can filter the retrieval set by the requesting user's permissions at query time. A fine-tuned model has absorbed every document into its weights and cannot un-know one for a particular user. There is no fix for this short of a model per permission set.

Attribution. RAG can cite the source passage, which is what makes an answer verifiable and is usually a hard requirement for internal knowledge tools.

Fine-tuning teaches behaviour, format and style, not facts. It is the right tool for "always respond in this JSON structure" or "adopt this domain's tone and conventions" — and it composes with RAG rather than competing with it.

Putting everything in the context window fails on cost, latency and the well-documented degradation of long-context retrieval; 50,000 documents will not fit in any case.

What actually determines quality

Almost all RAG failures are retrieval failures. If the right passage is not in the context, no model recovers.

Chunking. Fixed-size splits cut tables from their headers and clauses from their conditions. Structure-aware chunking with overlap, and keeping document metadata on every chunk, does more for quality than any embedding upgrade.

Hybrid retrieval. Dense vectors miss exact identifiers — product codes, error numbers, policy references — that BM25 keyword search finds immediately. Running both and fusing the results is consistently the single largest quality improvement available.

Reranking. Retrieve 50 candidates, rerank with a cross-encoder, pass the top 5. Usually a larger gain than switching embedding models.

Query rewriting. Users ask follow-up questions with pronouns. "What about the second one?" embeds to nothing useful. Rewriting against conversation history before retrieval is essential in a chat interface.

Evaluation. A held-out set of real questions with known correct sources, measuring retrieval recall separately from answer quality — so you know which half is failing.

What a strong answer adds

Naming the architecture around the model: an AI gateway for cost control and provider failover, prompt versioning so a prompt change is a reviewable deployment, and logging of retrieved context alongside answers, since a bad answer cannot be diagnosed without knowing what the model was shown.