Conversational Query Rewriting

"What about for Postgres?" retrieves nothing useful. The retriever is stateless and the query is meaningless in isolation: the subject was established three turns earlier and this turn contains a pronoun-like reference, an ellipsis, and no content words that identify the topic. Conversational rewriting converts it into "how do I configure connection pooling for Postgres", which a retriever can serve.

This is the single most impactful component in conversational RAG, and it is frequently omitted in favour of embedding the raw follow-up or concatenating the whole history, both of which fail in specific ways.

What the rewrite has to resolve

Four linguistic phenomena account for most of the difficulty. Coreference: "it", "they", "that one" refer to entities in earlier turns. Ellipsis: "and for larger datasets?" omits the verb and object entirely. Topic carryover: the subject persists implicitly across turns without being renamed. Topic shift: at some point the user changes subject, and carrying context forward becomes actively harmful.

The fourth is what makes naive approaches fail. Concatenating the full history into the query means every retrieval is polluted by whatever was discussed earlier, and the pollution grows with conversation length. A rewriter must decide not only what to carry forward but when to stop.

Three implementations

Generative rewriting prompts or fine-tunes a model to produce a self-contained query from the history and the current turn. It handles all four phenomena and is the standard approach. It costs a model call on the critical path, which is typically 200 to 600 milliseconds before retrieval can even begin, and it can hallucinate constraints the user never stated, narrowing retrieval to a subtopic the conversation never touched.

Term selection classifies which terms from the history to append, framing the task as extraction rather than generation. It is faster, cannot hallucinate, and handles ellipsis poorly since there is often no term to select for a missing verb.

Conversational dense retrieval skips rewriting and encodes the history directly into a query embedding, trained end-to-end on conversational relevance data. It removes the latency of a separate generation step and the interpretability along with it: when retrieval fails there is no intermediate rewrite to inspect, which is a real operational cost.

The hybrid that appears in most production systems keeps the generative rewrite for its quality and interpretability, and caches aggressively, since the rewrite depends only on the conversation prefix.

Evaluating it

The trap is evaluating the rewrite against a reference rewrite by string similarity. Two rewrites can be lexically different and retrieve identically, and one word's difference can change retrieval completely. The measurement that matters is downstream: retrieval quality using the rewritten query, compared against retrieval using a human-written self-contained query as the ceiling. QReCC and similar conversational collections provide the human rewrites needed to establish that ceiling.

When it breaks

Long conversations degrade rewriting. Twenty turns of history exceed what a rewriter attends to reliably, and topic shifts accumulate. Windowing over recent turns plus a running summary is the standard structure, and it introduces its own decision about what the summary retains.

The rewrite hides the failure. When retrieval goes wrong, the visible symptom is a bad answer, and the cause is often a rewrite that dropped a constraint or invented one. Logging the rewrite alongside the retrieved documents is what makes this debuggable, and it is routinely omitted.

Multi-intent turns do not rewrite into one query. "How does it compare to MySQL, and what about licensing?" is two retrievals. A single rewrite either serves one intent or produces a blend that serves neither, and handling it properly means decomposition rather than rewriting.

Rewriting for retrieval is not rewriting for generation. The retriever wants keyword-rich, self-contained text; the generator needs the original conversational framing to answer in context. These are different artefacts, and collapsing them costs quality on one side or the other.