Query Rewriting and Expansion for Better Retrieval

The raw query is not the question

Watch what real users type into a retrieval box and the gap between the query and the question is immediate. They enter three words where the answer needs six concepts. They use the internal nickname for a system, not the name that appears in the documentation. They ask "how do I fix this" with no antecedent because the "this" was on their screen a moment ago. The pipeline treats that fragment as a faithful statement of intent, embeds it, and searches. The embedding of an underspecified fragment points at a vague region of vector space, and the retriever returns passages that are plausibly near it and rarely the one that answers the question.

The instinct is to blame retrieval and reach for a better embedding model or a re-ranker. Those help, but they are answering the query as asked. If the query itself is a poor expression of the need, a stronger retriever just finds better matches for the wrong thing. The higher-leverage move sits one step earlier: transform the query before retrieval so that what reaches the index is a fuller, less ambiguous, more document-shaped version of the user's intent. An LLM is well suited to this because it can expand, disambiguate and restructure text cheaply.

This is the least-built stage in most RAG systems. Teams invest heavily in chunking, embeddings and reranking, then feed all of it a raw string straight from the input box. Query rewriting is not one technique but a family of them, each aimed at a distinct way that raw queries fail. The sections below walk through expansion, decomposition, multi-query, HyDE and conversational rewriting. None is universal; the skill is diagnosing which failure you have before spending latency to fix it.

Query expansion

Query expansion adds terms to the query so retrieval catches documents the literal wording would miss. A user asks about "firing an employee"; the policy that answers them is titled "involuntary separation procedures" and never once uses the word "firing." Dense retrieval bridges some of that gap through meaning, but not all of it, and a lexical leg of a hybrid search catches nothing at all when no content word overlaps. Expansion closes the gap by appending synonyms and related terms, so the enriched query carries both the user's vocabulary and the corpus vocabulary and matches on whichever the document happens to use.

An LLM does this well because it knows the paraphrases and the domain-adjacent terms a keyword thesaurus would miss. Given "firing an employee" it can add "termination, dismissal, involuntary separation, offboarding," and the retriever now has several routes to the right passage. Expansion is most valuable exactly where user language and document language diverge: consumer-phrased questions against formal corpora, plain terms against legal or clinical writing, and acronym-versus-expansion mismatches.

The failure mode is over-expansion. Every term you add widens the query, and a query that means too many things retrieves documents about all of them. Push it far enough and precision collapses under a drift of loosely related passages, which is worse than the narrow miss you started with. Expansion helps when the problem is vocabulary mismatch on an otherwise clear question; it hurts when the query was already specific, because there was nothing to bridge and the extra terms only add noise.

• Expansion earns its keep when: the query is short and clear but uses different words than the corpus, jargon and plain language collide, or a hybrid pipeline's lexical leg is starved for matching terms.
• Expansion backfires when: the query already names the exact concept, the added terms pull in adjacent-but-wrong topics, or an identifier like a part number gets buried under vague synonyms.

Query decomposition

Some questions are not one question. "How does our parental leave policy compare to our sick leave policy, and which one covers a newborn adoption?" bundles three retrieval needs into a single string: the parental leave rules, the sick leave rules, and the adoption provision. Embedding that whole sentence produces a vector that is the average of three intents and a good match for none. The passage on parental leave and the passage on adoption sit in different regions of the space, and a single query cannot be near both at once.

Decomposition breaks the compound question into sub-queries, retrieves for each one separately, then combines the results into a single context set for the generator. Each sub-query is now focused, so each retrieval is sharp, and the union of their results covers every part the user asked about. An LLM performs the split, which is a natural fit: identifying that a question has distinct parts and phrasing each part as a standalone query is exactly the kind of restructuring language models do reliably.

The payoff is recall on questions that a single embedding would only ever half-answer. It is the right tool for comparisons ("A versus B"), multi-hop questions where one answer feeds the next, and any request that visibly enumerates several things. It is the wrong tool for a simple, atomic question, where splitting invents sub-parts that do not exist and multiplies retrieval cost for nothing. Decomposition also compounds latency, since sub-queries often run as separate retrievals, so reserve it for questions that are genuinely compound rather than applying it to every input.

Multi-query

Even a single, clear intent can be phrased several ways, and the phrasing you happened to use may not be the one nearest the answer in vector space. Embedding is sensitive to surface form; "reset my password," "recover a locked account," and "I can't log in" express one need but land in three slightly different places. If the document that answers them is closest to a phrasing the user did not choose, a single-query retrieval misses it for no deeper reason than word choice. This is a coverage gamble, and multi-query stops gambling.

The technique asks an LLM to generate several alternative phrasings of the same question, runs a retrieval for each, and unions the results before ranking. Where any one phrasing might miss the target document, the set of phrasings collectively covers more of the space around the intent, so the right passage is far likelier to appear in at least one result list. The fan-out hedges against the brittleness of a single embedding.

 USER QUERY: "reset my password"
      |
      | LLM generates variant phrasings
      v
   +--> "how do I recover a locked account"  --> [d4, d9, d2]
   +--> "steps to change a forgotten password" --> [d9, d1, d7]
   +--> "unlock account after failed logins"   --> [d5, d9, d3]
                       |
                       v
        UNION + dedupe --> {d9, d4, d2, d1, d7, d5, d3}
                       |
                       v
        fuse / re-rank --> top-k to the generator
            

Multi-query trades cost for recall, and the arithmetic is honest about it: one rewrite call plus several retrievals per user question, then a merge step (reciprocal rank fusion is the usual choice, since it combines ranked lists without calibrating score scales). It shines when phrasing sensitivity is your failure mode and paraphrase-heavy traffic keeps missing good documents. It is overkill for a corpus and query mix where a single well-formed query already retrieves reliably, where the extra retrievals buy little beyond a larger bill and more latency.

HyDE: embed a hypothetical answer

The other techniques rewrite the question into a better question. HyDE, or hypothetical document embeddings, does something stranger: it stops embedding the question at all. The observation behind it is that a question and its answer do not look alike. A query is short, interrogative and sparse; the passage that answers it is longer, declarative and dense with the very terms the query omits. Embedding a question and searching for nearby documents means searching with a shape that does not resemble what you are trying to find.

HyDE asks the LLM to first write a hypothetical answer to the query, a plausible passage of the kind the corpus might contain, then embeds that generated passage and uses its vector for retrieval. The hypothetical answer, being answer-shaped, sits closer in embedding space to the real answer than the original question did. It does not matter that the generated text may be factually wrong or partly invented; it is never shown to the user and never enters the final context. It serves only as a retrieval probe whose vector points at the right neighborhood.

Presented plainly, this is one technique among several, not a default, and it comes with real caveats. It adds a full generation step before retrieval even begins, so it is among the more expensive rewrites in both latency and cost. Its quality depends on the model producing a hypothetical that resembles a genuine corpus document; on a highly specialized domain where the model has weak priors, the fabricated answer can point at the wrong region and mislead retrieval rather than sharpen it. HyDE tends to help most on open-ended, knowledge-style questions and to help least where the query already contains a precise identifier that a literal match would have nailed. Treat it as a candidate to test on a held-out set, not a technique to switch on by faith.

Rewriting conversational queries

Chat interfaces break retrieval in a specific, predictable way. A user asks "what is our policy on remote work?", gets an answer, then types "what about its cost?" That second message is a perfectly natural human utterance and a useless retrieval query. "Its" has no referent inside the string; embedded alone, "what about its cost" retrieves generic passages about costs and nothing about remote work. The intent lives in the conversation history, not in the message the user just sent, and the retriever only sees the message.

Conversational rewriting resolves this by folding the relevant history into the current turn to produce a standalone query. The mechanism is coreference resolution: an LLM reads the prior turns, resolves "its" to "remote work policy," and rewrites the follow-up as "what is the cost of the remote work policy?" That standalone query carries its full intent in one string and retrieves correctly on its own. For any multi-turn RAG system this is not an enhancement but a prerequisite; without it, retrieval quality falls off a cliff the moment a user asks a second, context-dependent question, which they almost always do.

The discipline is deciding how much history to fold in and when. Too little context and the rewrite misses the referent; too much and the model drags in stale topics from earlier turns and rewrites toward a question the user has moved on from. Rewrite only when the current turn actually depends on prior context, and keep the window tight. The table below places the techniques from this article side by side, with the failure each addresses.

The architect view

The governing principle is simple: the string a user submits is an input to be improved, not a specification to be honored. Rewriting it before retrieval is one of the cheapest quality levers in a RAG system, and it stays underused because it lives in the seam between the interface and the index, where nobody owns it by default. Put a rewriting stage there deliberately, and treat it as a first-class part of the pipeline rather than a preprocessing afterthought.

Do not switch on every technique. Each targets a specific failure, and applying the wrong one degrades results: expansion muddies an already-specific query, decomposition invents sub-parts of an atomic question, HyDE misfires on a domain the model does not know. Read your failing queries first. Vocabulary mismatch calls for expansion, compound questions for decomposition, paraphrase brittleness for multi-query, question-answer shape gaps for HyDE, and multi-turn chat for conversational rewriting, which is close to mandatory the moment you ship a chat surface at all.

Every rewrite is at least one extra LLM call before retrieval, so it is latency and cost you are adding to the hottest path in the system, and a rewrite can drift from the user's actual intent. Budget it. Cache rewrites for repeated queries, use a small fast model for the rewrite where quality allows, and reserve the expensive fan-out techniques for the traffic that needs them rather than the whole stream. Above all, measure the lift on a held-out set the way you would any retrieval change; a rewrite that does not move recall or precision is pure overhead.

Rewriting sits upstream of the rest of the retrieval stack and multiplies it. A better query feeds a stronger signal into hybrid search, and the vocabulary-mismatch problem expansion patches is the same one that domain-adapted embeddings address at the model level, so weigh which layer to fix. Validate the whole chain with a golden set as covered in the RAG evaluation deep-dive, and watch a rewrite reshape the results live in the RAG Explorer. Fix the query, and every stage downstream gets easier.