RAG vs Long Context

When context windows hit a million tokens, a tempting simplification appeared: skip the retrieval pipeline, its chunking bugs and its embedding drift, and just put the whole corpus in the prompt. Whether that is wise turns out to be a question with measured answers, and they are less one-sided than either camp predicted. A controlled comparison across question-answering datasets using Gemini 1.5 Pro, GPT-4o, and GPT-3.5-Turbo found that when the budget allows it, feeding the full long context consistently outperforms RAG on average quality, while RAG retains a decisive cost advantage (Li et al., 2024, Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach, EMNLP Industry, arXiv:2407.16833).

Why each side wins where it does

Full context wins on questions whose evidence is diffuse: multi-hop chains, summarisation-shaped questions, anything where relevance is only visible once you have read everything. A retriever has to guess relevance before the model reasons, from lexical or embedding similarity alone, and multi-hop questions punish that guess: the second hop's evidence often shares no surface features with the query. Long context defers the relevance decision to the model itself, which is a better judge, at quadratic-attention prices.

RAG wins on cost by one to two orders of magnitude, wins on latency for large corpora, and wins outright when the corpus exceeds any window, changes faster than you can re-prompt, or requires per-user access control, which a monolithic prompt cannot express. Follow-up benchmarks complicate the quality story too: LaRA's controlled comparison across four QA task types found the winner flips with model strength, context length, and question type, with weaker models often better served by RAG and stronger ones by long context (Li et al., 2025, LaRA, arXiv:2502.09977). "Long context beats RAG" is a statement about frontier models on corpora that fit; it does not survive generalisation.

Self-Route: making the choice per query

The useful observation in Li et al. is that the two approaches agree on most queries; the expensive disagreements cluster on a minority. Their Self-Route method exploits this: run RAG first, and let the model itself declare whether the retrieved chunks suffice to answer; if it declines, escalate that query to the full context. Because most queries are answerable from retrieval, the expensive path runs rarely, cutting cost dramatically while tracking full-context quality. The general lesson outlives the specific method: RAG versus long context is a per-query routing decision, not an architecture commitment, and the model's own abstention signal is a workable router.

The hybrid that production actually converges on

Deployed systems increasingly refuse the dichotomy. Retrieval selects a generous candidate set, the long window absorbs it whole (removing chunking's worst failure, evidence split across a chunk boundary), and prompt caching amortises any shared prefix. Long context also changed retrieval's job description: recall at 50 with a wide window beats precision at 5 into a narrow one, so the retriever is tuned as a coarse filter rather than a sniper (see retrieval-augmented generation).

When it breaks

• Long context is not immune to its own marketing. Effective length lags advertised length, degradation is non-uniform, and stuffing 500k tokens into a model that reads 64k well produces confident answers sourced from noise (see context rot).
• RAG failure is silent; long-context failure is expensive. A retriever that misses the evidence produces a fluent wrong answer with no error signal. Full context fails louder in the bill and the latency budget instead.
• Routing inherits the router's calibration. Self-Route trusts the model to know when retrieval sufficed; on questions where the model is confidently wrong about sufficiency, the cheap path answers wrongly and the escalation never fires.
• Caching discipline decides the economics. Full-context costs assume the corpus prefix is cached; a corpus that updates hourly invalidates the cache hourly, and the comparison swings back toward RAG.