Agentic and Iterative Retrieval

"Which university employed the author of the paper that introduced the Conformer?" One retrieval cannot answer this. Embedding the question and searching returns documents about universities, or about Conformers, and rarely the one document that connects them, because the second search term is not knowable until the first search has returned. Classic RAG issues exactly one query, reads the results, and generates. For any question whose retrieval target depends on an intermediate conclusion, that architecture has already lost before generation starts.

The fix is to make retrieval a loop rather than a prefix. Three distinct designs do this, and they are often conflated.

Interleaving retrieval with reasoning

IRCoT alternates the two: generate one sentence of chain of thought, use that sentence as the next retrieval query, add what comes back to the context, generate the next sentence (Trivedi et al., 2023, Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions, arXiv:2212.10509). The reasoning steers retrieval and the retrieved text steers reasoning. On multi-hop benchmarks including HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC the authors report retrieval gains of up to 21 points and downstream QA gains of up to 15 points over a one-step retrieve-and-read baseline with the same model.

Note what is being fixed. The generator was never the bottleneck on these questions; the retriever was, and it was starved because nobody could write its second query in advance.

Learning when to retrieve at all

Self-RAG takes a different angle: train the model to decide, token by token, whether retrieval is warranted, and to critique what comes back (Asai et al., 2024, Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, arXiv:2310.11511). The model emits special reflection tokens interleaved with normal output: a retrieve-or-not decision, then judgments on whether a retrieved passage is relevant, whether the generated sentence is supported by it, and whether the output is useful. These are ordinary vocabulary items trained with next-token prediction, so no separate classifier is needed, and at inference they become controls: weight the decoding to favour supportedness and you get a more conservative, better-cited system without retraining.

The unconditional-retrieval habit this replaces is worth naming. Retrieving for every query injects irrelevant passages into arithmetic questions and creative requests, where they measurably degrade output. Deciding not to retrieve is a capability.

Search agents

The third design gives a tool-using agent a search tool and lets it run its own loop: query, read, refine, query again, stop when satisfied. This is what "deep research" products do. It subsumes the previous two designs and inherits every problem of long-horizon agents (see long horizon agent reliability): per-step reliability compounds, the context fills with search results the agent must now re-read, and the stopping decision is itself a judgment the model can get wrong in both directions.

When it breaks

• Latency multiplies by the number of hops. A four-hop trace is four sequential retrieve-and-generate rounds, each with a full prefill. Interactive latency budgets rarely survive this, which is why iterative retrieval shows up in research and reporting products before it shows up in chat.
• Errors compound in the direction of the first mistake. A wrong first-hop retrieval produces a wrong intermediate conclusion, which becomes the next query, which retrieves confidently wrong support for it. Single-pass RAG fails obviously; iterative RAG fails coherently, which is worse.
• Context accumulates fast. Every hop appends passages that the model must re-read on the next hop, so token cost grows roughly quadratically in the number of hops unless you actively evict or summarise.
• Stopping is unsolved. Stop too early and the answer is unsupported; stop too late and cost runs away with no accuracy gain. Fixed hop caps are crude but are what most production systems actually use.
• Evaluation must be per-hop. End-to-end accuracy hides whether the loop failed at hop one or hop four, and those have opposite fixes. Log the query issued at each hop and the passages returned, and score retrieval separately from generation.