Query Transformation for Retrieval

Retrieval assumes the query and the answer look alike in embedding space. Users routinely violate that assumption. They ask "does it cover this?" with the antecedent three turns back in the conversation. They ask "why is our churn up in EMEA and did the pricing change cause it", which is two questions whose answers live in different documents. They ask short questions whose answers are long, and a nine-word query embedded into the same space as a 400-word passage is a genuinely awkward comparison.

Query transformation is the stage that rewrites the user's input into one or more things that retrieve well. It runs before the retriever and costs one small LLM call.

Four transformations that matter

Contextual rewriting. In a multi-turn chat, resolve the query against the conversation into a standalone question. "What about in Germany?" becomes "What is the VAT registration threshold in Germany?" Without this step, a follow-up embeds to nearly nothing. This single transformation is the highest-value one in any conversational RAG system and is often the only one worth having.

Decomposition. Split a compound question into sub-questions, retrieve for each, and merge the evidence. Comparative and multi-hop questions ("how does our refund policy differ from the 2024 version") cannot be answered by one retrieval, because no single passage contains both sides. Related to prompt chaining, and it is what turns a RAG pipeline into something closer to an agent.

Multi-query fan-out. Generate three to five paraphrases of the query, retrieve for each, and fuse the ranked lists with reciprocal rank fusion (Cormack, Clarke & Büttcher, SIGIR 2009, Reciprocal Rank Fusion):

with \(k \approx 60\). Fan-out covers vocabulary the original phrasing missed. It multiplies retrieval cost by the number of variants, which is cheap, and multiplies reranking cost, which is not.

HyDE. Ask the model to write the answer it imagines, then embed that hypothetical document and search with it (Gao et al., 2022, Precise Zero-Shot Dense Retrieval without Relevance Labels, arXiv:2212.10496). The insight is that document-to-document similarity is an easier comparison than question-to-document: the hypothetical answer shares length, register and vocabulary with the real one. HyDE was strongest as a way to get good zero-shot retrieval without labelled training data; against a well-tuned domain retriever the gain narrows.

Choosing between them

When it breaks

Every transformation is an LLM call in front of the retriever, which means it can hallucinate, add latency, and fail in ways that are invisible downstream. A rewriter that turns "the Q3 number" into "the Q3 revenue number" when the user meant headcount has corrupted the query, and the pipeline will retrieve confidently against the wrong question.

Three specific traps:

• Precision loss on exact terms. Rewriting can drop a part number, a version string, or a name. Preserve rare tokens verbatim, or always run the original query alongside the rewritten one and fuse.
• Latency stacking. Rewrite, fan-out, retrieve, rerank, generate is five sequential stages. Multi-query fan-out at least parallelises; decomposition often cannot.
• Compounding on bad input. HyDE against a corpus the model knows nothing about produces a hypothetical answer full of plausible fiction, which then retrieves documents that match the fiction.

Measure transformations end to end, on answer quality with the same eval set used for every other retrieval change. Retrieval metrics alone will often show a gain from fan-out that the generator cannot use.