Query Rewriting & HyDE

Introduction

The last lesson fixed retrieval after the fact — rerank the candidates you got back. This lesson works the other end: improving the query itself, before it ever hits the index. Because here's a truth naive RAG ignores — the user's raw query is often a bad search key.

It might be conversational ("what about after that?" — meaningless without history), vague, or — most subtly — phrased nothing like the documents that answer it. A short question and a detailed answer can be genuinely related yet land far apart in embedding space. Query transformation rewrites or reshapes the query to close that gap.

In this lesson you'll learn:

Query rewriting — turning vague/conversational queries into clear, standalone ones
HyDE — the elegant trick of searching with a hypothetical answer
The query–answer gap (made visual), and where these help vs. hurt
The discipline that separates pros from cargo-cultists: name the failure before you add a transform

Where This Fits: Before Retrieval, Not After

Keep the pipeline picture straight, because it organizes the whole advanced-retrieval section:

raw query → ✏️ TRANSFORM → 🔍 retrieve → 🎯 rerank → 🤖 LLM

Query transformation (this lesson) acts before retrieval — it changes what you search with.
Reranking (last lesson) acts after retrieval — it changes the order of what you got back.

They're complementary and stack. And there's a deep reason both exist: the bi-encoder that powers first-stage retrieval (L98) embeds the query in isolation — so if the query is a poor representation of the user's intent, retrieval starts from a bad place. Reranking can't fully recover from that (the recall ceiling). Fixing the query attacks the problem at its source.

An infographic titled 'Query Transformation — Fix the Query Before You Search'. A positioning bar shows the pipeline: raw query, then TRANSFORM (this lesson, BEFORE retrieval, highlighted), then retrieve, then rerank (last lesson, AFTER), then LLM; a note says the raw user query is often a bad search key — vague, conversational, or phrased nothing like the answer — so fix it first. A toolbox of techniques: Rewrite to standalone (an LLM turns a vague or conversational follow-up into a clear self-contained query by resolving pronouns and ellipsis, e.g. 'what about after that?' becomes 'return policy after 30 days?'; essential for chat); Expand or step-back (add synonyms and related terms, or generate a broader more abstract question to pull in high-level context); HyDE (generate a hypothetical answer — facts don't matter — embed that and search with it, bridging the question-to-answer gap; zero-shot, strong out-of-domain); and Decompose or multi-query (split a compound question into sub-queries or fan out N variants — next lesson). A HyDE mechanism strip: query, then the LLM writes a fake answer, then embed the fake answer, then retrieve real docs near it; a question and its answer look different in vector space, so a fake answer lands beside the real answers, and the hypothetical can be wrong because the encoder's bottleneck keeps the semantic shape and discards the made-up facts. A caveat: every transform adds an LLM call before retrieval (latency and cost); HyDE adds 25 to 40 percent latency, can drift if it hallucinates wrong (average several to stabilize), and strong embedders shrink the gain; the rule is to add a transform only when you can name the retrieval failure it fixes, otherwise it's expensive theatre. Banner: reranking fixes results after retrieval, query transformation fixes the query before; reshape the query to look like the answer.

Query Rewriting: Make the Query a Good Search Key

The most common and highest-ROI transformation. Use an LLM to rewrite the raw query into a better one before searching. The dominant use case — and the one that breaks most chatbots if you skip it — is conversational decontextualization: turning a context-dependent follow-up into a standalone query.

Consider a multi-turn chat. The user asks "What's the return policy for electronics?", then follows with "what about after that?" That follow-up, embedded on its own, retrieves nothing useful — it has no nouns, no topic. Feeding the entire chat history into the retriever is noisy and dilutes the signal. The fix is to rewrite the follow-up into a self-contained question using the history:

# Conversational follow-ups are NOT standalone search queries. Rewrite first.
REWRITE_PROMPT = """Given the chat history and the user's latest message, rewrite the
message into a single, standalone search query. Resolve pronouns and fill in implied
context. Output ONLY the rewritten query.

History:
{history}

Latest message: {message}
Standalone query:"""

def rewrite_query(history, message):
    return llm(REWRITE_PROMPT.format(history=history, message=message)).strip()

# history: "User: What's the return policy for electronics? / AI: 30 days, unopened."
# message: "what about after that?"
rewrite_query(history, "what about after that?")
# → "What is the return policy for electronics after the 30-day window?"   ← now retrievable

The rewrite does coreference resolution (replace pronouns with the actual entities) and ellipsis handling (fill in the implied "return policy" and "30-day window"). Beyond conversation, rewriting also covers spelling/normalization, acronym expansion, and query expansion (adding synonyms). A cousin, step-back prompting, instead generates a broader, more abstract question to retrieve high-level background before answering a specific one.

But heed the warning: rewriting is the cheapest transform and the easiest to overuse. If the query is already specific and clear, an LLM rewrite just adds latency and cost without adding signal — and can even distort a good query. Rewrite to fix a named problem (conversational refs, vagueness), not by reflex.

The Query–Answer Gap (and HyDE's Insight)

Now the subtle failure that rewriting alone doesn't fix. Embeddings place text by meaning and style — and a question is stylistically very different from its answer:

Query: "forgot my login" — terse, interrogative, sparse.
Answer doc: "To reset your password, open the login screen and click 'Forgot password'; a reset link is emailed and expires after 60 minutes." — declarative, detailed, specific.

These can sit in different regions of embedding space, so a question's nearest neighbours are often other questions or vaguely-related text, not the actual answer. This is the query–answer gap, and it's exactly what L95's index-time hypothetical questions attacked from the other side.

HyDE (Hypothetical Document Embeddings) — Gao et al., 2022, "Precise Zero-Shot Dense Retrieval without Relevance Labels" — attacks it from the query side with a delightfully counterintuitive move:

Don't embed the question. Ask an LLM to generate a hypothetical answer to it — even if the answer is made up — then embed that and use its vector to search.

Because the hypothetical answer is written in the style and vocabulary of a real answer, its embedding lands in answer-space, right next to the documents that genuinely answer the question. Toggle the interactive below and watch the search probe jump from "question-space" (retrieving a weak match) to "answer-space" (retrieving the real answer):

Toggle the search probe: the raw question sits in 'question-space' near a weak match; a HyDE hypothetical answer lands in 'answer-space' beside the real answer. Watch which doc gets retrieved flip.

The part that trips people up: "but the generated answer might be wrong!" That's fine — you never show it to the user. It's purely a search probe. As the HyDE paper puts it, the encoder's dense bottleneck preserves the semantic shape of a relevant answer while discarding the specific (possibly hallucinated) facts. You retrieve real documents; the fake answer is thrown away.

HyDE in Code

The whole method is a few lines: generate → embed → search. Note the optional n > 1 to average several hypothetical answers, which stabilizes against a single bad generation:

# HyDE: search with a hypothetical ANSWER, not the raw question.
HYDE_PROMPT = "Write a short passage that directly answers this question:\n{q}\n\nPassage:"

def hyde_retrieve(query, k=10, n=1):
    # 1) generate n hypothetical answer(s) — factual accuracy does NOT matter
    hypotheticals = [llm(HYDE_PROMPT.format(q=query)) for _ in range(n)]
    # 2) embed them (average for stability) and search with THAT vector
    vecs = embed(hypotheticals)                 # answer-shaped → lands near real answers
    probe = vecs.mean(axis=0)
    return vectorstore.search(probe, k=k)        # returns REAL docs (the hypothetical is discarded)

# The fake answer may contain wrong facts — fine. We only use its EMBEDDING as a search
# probe; the encoder's bottleneck keeps the semantic "shape" and drops the invented details.

HyDE is zero-shot — no training, no labels — and the paper showed it beating the strong unsupervised retriever Contriever and rivaling fine-tuned retrievers, including in non-English settings. Its sweet spot is exactly where dense retrieval is weakest: new, unseen domains and specialized corpora (medical, legal, developer support) where off-the-shelf embeddings haven't learned the vocabulary.

The Honest Take: Don't Add Transforms as Theatre

Query transformations are powerful, and also the most over-applied techniques in RAG. The expert discipline is knowing when not to use them.

The costs are real:

Every transform adds an LLM call before retrieval — latency on the critical path and dollars, on every query.
HyDE specifically adds a full generation step (one 2026 study measured +25–40% response time) and can drift: if the LLM hallucinates in the wrong direction, you embed a misleading answer and retrieve worse documents than the raw query would have. Averaging multiple hypotheticals mitigates this but costs even more.
Modern strong embedding models shrink the payoff. If your embedder already gets high recall on plain queries, HyDE (and aggressive rewriting) add complexity for little gain — the same lesson we hit with hybrid search and semantic chunking.

The rule that separates engineers from cargo-cultists: Don't add a query transformation until you can name the specific retrieval failure it fixes — and the latency you're willing to pay for it. A transform added "because the architecture diagram looked lonely" is expensive theatre.

So: conversational app? Query rewriting is near-mandatory. Zero-shot / out-of-domain / big query–answer gap? Try HyDE and measure recall before vs. after. Already-specific queries on a strong in-domain embedder? You may need none of this. And when different queries need different treatment, you route them — which is the adaptive retrieval routing lesson (L101).

🧪 Try It Yourself

Drive the HyDE widget, then diagnose real cases:

In the interactive, search with the raw query, then toggle to HyDE. Which document does each retrieve? Why does the question land near a weak match while the hypothetical answer lands near the real one?
The hypothetical answer HyDE generates contains a wrong fact. Does that necessarily ruin retrieval? Why or why not?
A user in a chat asks "and the international one?" Retrieval returns garbage. Which transform fixes this, and what does the rewritten query look like?
Your queries are already long, specific, in-domain product questions, and your embedder has high recall. A teammate wants to add HyDE "to be safe." What's your response?
You add HyDE and p95 latency jumps 35% while recall barely moves. What does that tell you, and what do you do?

→ (1) Raw query → a weak/wrong doc (it sits in question-space, near generic text); HyDE → the real answer (the hypothetical answer sits in answer-space). The gap is stylistic: questions and answers are phrased differently. (2) No — HyDE uses only the answer's embedding as a search probe; the encoder's bottleneck keeps the semantic shape and discards the specific facts, and you retrieve real docs. (Drift only hurts if the answer is wrong in a way that changes its overall topic/shape.) (3) Query rewriting (decontextualization) → "What is the international return/shipping policy?" (resolve the ellipsis from history). (4) Push back: HyDE shines when there's a query–answer gap or out-of-domain text; on specific in-domain queries with a strong embedder it adds latency/cost for little gain — measure before/after, don't add by reflex. (5) It's not fixing a real failure for your workload (no recall lift) — remove it. That's the "expensive theatre" anti-pattern; transforms must earn their latency.

Mental-Model Corrections

"The user's query is the search query." Often it's a poor search key — conversational, vague, or stylistically unlike the answer. Reshape it first.
"HyDE's generated answer must be accurate." No — it's a search probe, never shown to the user. The encoder keeps its semantic shape and drops the (possibly hallucinated) facts; you retrieve real documents.
"HyDE = the index-time hypothetical questions from L95." Opposite ends: HyDE makes a hypothetical answer at query time; L95 stored hypothetical questions at index time. Both bridge the query–answer gap.
"Query rewriting and reranking do the same job." Rewriting acts before retrieval (changes the search key); reranking acts after (reorders results). Complementary.
"More query transformations = better retrieval." Each adds latency + cost, and HyDE can drift. Name the failure mode first, or it's theatre.
"These help every system." Strong in-domain embedders shrink the gain; conversational apps need rewriting most. Measure on your data.

Key Takeaways

Query transformation fixes the query before retrieval (vs reranking, which fixes results after). The raw user query is frequently a bad search key.
Query rewriting turns vague/conversational follow-ups into standalone queries (coreference + ellipsis) — essential for chat. Also covers expansion and step-back prompting. But it's the easiest transform to overuse — don't rewrite an already-clear query.
The query–answer gap: questions and answers are phrased differently and land in different embedding regions. HyDE (Gao 2022) bridges it by generating a hypothetical answer, embedding that, and searching with it — the made-up facts don't matter, only the semantic shape. Zero-shot, strong out-of-domain.
Honest costs: transforms add an LLM call before retrieval (latency/cost); HyDE adds ~25–40% latency and can drift (average several to stabilize); strong embedders shrink the gain.
The rule: add a transform only when you can name the retrieval failure it fixes — otherwise it's expensive theatre. Measure before/after.
Next: multi-query & query decomposition — when one reshaped query isn't enough and you need to fan out or break the question apart.