Multi-Query & Query Decomposition

Introduction

Last lesson, we reshaped a query into one better query. But sometimes one query — however well-crafted — simply can't do the job, and you need several. This lesson covers the two techniques for that, and the single most important thing is to not confuse them, because they solve opposite problems:

Multi-Query (fan-out): the user asked one question, but a single phrasing of it might miss relevant documents. So you generate several rephrasings of the same question and pool the results → a recall boost.
Query Decomposition (split): the user asked one question that is really several questions in a trenchcoat. So you break it into different sub-questions, answer each, and compose → a coverage fix.

Both end up running multiple retrievals, which is why they live here — and both lean on RRF (L97) to fuse. In this lesson:

Multi-query fan-out — and why rephrasing boosts recall
Decomposition — parallel vs sequential/multi-hop
The crisp recall-vs-coverage distinction (made visual)
The cost — and why you should route simple queries past both

An infographic titled 'Multi-Query vs Decomposition — Fan Out or Split'. It says: when one query isn't enough, generate several, but for two very different reasons. Left column, Multi-Query (fan-out): SAME question, many phrasings; the flow is query, then 3 paraphrases, then retrieve each, then fuse with RRF; result sets OVERLAP and their union recovers docs the original phrasing missed, so it boosts RECALL; for ambiguous or narrow queries and vocabulary mismatch, via LangChain MultiQueryRetriever. Right column, Decomposition (split): ONE compound question, many sub-questions; the flow is complex Q, then sub-questions, then answer each, then compose; sub-questions retrieve DIFFERENT (disjoint) docs, all needed to answer, giving full COVERAGE; for multi-part, multi-hop, cross-source questions, via LlamaIndex SubQuestionQueryEngine. A distinction strip: multi-query is same intent reworded (overlap to recall); decomposition is many intents, one per sub-question (disjoint to coverage); both emit multiple ranked lists which RRF fuses. Another strip: decomposition can be parallel (independent sub-questions, error-isolated, fast) or sequential / multi-hop (each depends on the previous answer, powerful but error cascades, shading into agentic RAG). A caveat: both run N retrievals plus LLM calls so latency and cost multiply; multi-query has diminishing returns after 3 to 4 variants; decomposition is heavy (an LLM planning step, can add seconds per query) and over-splits simple questions, so ROUTE simple queries past both (next lesson) and cache sub-queries. Banner: one query not enough? Fan it out for recall, or split it for coverage — and route simple queries past both.

Multi-Query: Ask It Several Ways

A dense retriever matches on one embedding of one phrasing — and that's a lossy probe. The user's exact words might not match how the answer is written in your corpus (the vocabulary-mismatch problem again). Multi-query fixes this by fanning out: an LLM rewrites the query into N alternative perspectives (LangChain's MultiQueryRetriever defaults to 3), each is retrieved, and the results are pooled.

from langchain.retrievers.multi_query import MultiQueryRetriever
from langchain_openai import ChatOpenAI

# One extra LLM call rewrites the query into N perspectives (default 3),
# retrieves for each, then UNIONs + dedupes the results.
retriever = MultiQueryRetriever.from_llm(
    retriever=vectorstore.as_retriever(),
    llm=ChatOpenAI(model="gpt-5.5"),
)
docs = retriever.invoke("how do I get my money back?")
# Under the hood it generates e.g.:
#   "refund process and timeline"  |  "how to return an item for a refund"  |  "reverse or dispute a payment"
# → each hits the index → results pooled. (In production, fuse the ranked lists with RRF.)

The key property — and the interactive will make it obvious — is that the variants' result sets OVERLAP but aren't identical. Each phrasing catches some of the same documents and a few the others miss, so their union is larger than any single query's hits. That's a direct recall improvement: you stop being at the mercy of how the user happened to phrase it. It also handles ambiguity — different rephrasings explore different facets of an unclear query.

Because you now have several ranked lists, you fuse them — and this is exactly what RRF (L97) was built for. RRF's consensus property shines here: a document that several rephrasings all surface is very likely relevant and gets pushed to the top.

Decomposition: Some Questions Are Several Questions

Multi-query assumes the question is one thing, just hard to phrase. Decomposition handles the opposite case: a compound or multi-hop question that no single retrieval can satisfy. Consider:

"Is the Pro plan cheaper than Enterprise, and does Pro include priority support?"

That's two questions — one about pricing, one about support — and they live in different documents. A single search for the whole sentence retrieves the pricing doc or the support doc, and answers half the question. The fix: split it into atomic sub-questions, retrieve and answer each, then compose the final answer.

# Decomposition: split a COMPLEX question into atomic sub-questions, answer each,
# then synthesize. (Conceptual — LlamaIndex SubQuestionQueryEngine automates this.)
def decompose_and_answer(question):
    subs = llm(f"Break this into the minimal independent sub-questions needed to "
               f"answer it. One per line.\n\nQ: {question}").splitlines()
    # e.g. ["What is the price of Pro vs Enterprise?",
    #       "Does the Pro plan include priority support?"]
    sub_answers = []
    for sq in subs:
        ctx = retrieve(sq)                       # each sub-question → its OWN (disjoint) docs
        sub_answers.append(generate(sq, ctx))
    return generate(question, context="\n".join(sub_answers))   # COMPOSE the final answer

# Sequential/multi-hop variant: a later sub-question depends on an earlier ANSWER
#   ("Who directed the highest-grossing film of 2010?" → find the film, THEN its director).
#   Powerful, but errors in early hops cascade.

The defining property — the mirror image of multi-query — is that the sub-questions retrieve DIFFERENT, disjoint documents. Each sub-question is a distinct information need, and you need all of their answers to respond. That's a coverage guarantee, not a recall boost. LlamaIndex's SubQuestionQueryEngine automates the decompose → route-each-to-a-source → synthesize flow, and can even route different sub-questions to different data sources.

See the Difference: Overlap vs Disjoint

This is the whole lesson in one widget. Toggle between the two modes and watch the shape of the retrieved sets:

Toggle Multi-Query vs Decomposition. Multi-query: one question, 3 phrasings → overlapping results, union recovers missed docs (recall). Decomposition: one compound question → different sub-questions → disjoint results, all needed (coverage).

Multi-Query: the three phrasings produce overlapping sets (notice D1/D2 recurring). Their union recovers documents (D3, D4) that the original phrasing missed — recall up, same single intent.
Decomposition: the two sub-questions produce disjoint sets (P1 = pricing, P2 = support). Neither alone answers the question; you need both — coverage, different intents.

Overlap → recall. Disjoint → coverage. If you remember one thing from this lesson, remember which technique produces which.

Parallel vs Sequential Decomposition

Decomposition comes in two flavors, and the difference matters for both power and reliability:

Parallel decomposition. The sub-questions are independent — "price?" and "support?" don't depend on each other — so you retrieve and answer them simultaneously. This is fast and error-isolated: a mistake on one sub-question doesn't corrupt the others.

Sequential / iterative decomposition (multi-hop). Each sub-question depends on the answer to the previous one. "Who directed the highest-grossing film of 2010?" must first find the film, then look up its director — you can't ask the second question until the first is answered. This unlocks genuine multi-hop reasoning, but it has a dangerous failure mode: error cascades — a wrong answer at hop 1 poisons every step after it.

Sequential decomposition is where retrieval starts to look like reasoning, and it's the gateway to agentic RAG (L106) — systems that iteratively decide what to retrieve next based on what they've learned. For now, prefer parallel decomposition when sub-questions are independent (most cases), and reserve sequential for genuinely multi-hop questions where you accept the cascade risk.

The Honest Take: Multiply Carefully, and Route

Generating N queries means running N retrievals and N+ LLM calls — the cost and latency multiply, often on the critical path. Use this power deliberately:

Multi-query: diminishing returns after ~3–4 variants. Beyond that, extra rephrasings mostly add duplicates, latency, and cost for negligible recall gain. The default of 3 is a sensible ceiling for most apps.
Decomposition is expensive. It adds an LLM planning step plus multiple retrieve-and-synthesize rounds — one study measured ~16.7s/query of overhead (≈19s with reranking), slower than naive RAG. Cache decomposed sub-queries (a repeated complex question can skip the planning call) to claw that back.
Don't over-decompose. Splitting a simple, single-fact question into sub-questions is the expensive-theatre anti-pattern from last lesson — it adds latency and new failure modes (bad sub-question → bad sub-answer → bad final) for no benefit.

The fix is routing. A lightweight classifier should look at each query and dispatch it to the minimally sufficient strategy: a simple lookup goes straight to plain retrieval; an ambiguous one gets multi-query; a genuinely compound one gets decomposition. Most queries need none of this. That adaptive routing is the next lesson — it's what turns this whole toolbox from "run everything always" into "run the right thing per query."

🧪 Try It Yourself

Toggle the widget, then classify real queries:

In Multi-Query mode, which documents does the union recover that the raw query missed — and why do the variant result sets overlap rather than being disjoint?
In Decomposition mode, why can't a single search answer the whole question? What property do the two sub-questions' result sets have?
For each query, pick the technique (or neither): (a) "reset password" (b) "how can I get my refund / money back / payment reversed?" (c) "Which of our 3 plans has the best uptime SLA and the lowest price?" (d) "Who is the CEO of the company that acquired our biggest competitor in 2023?"
A teammate sets multi-query to generate 10 variants "for maximum recall." Good idea?
Your decomposition pipeline is correct but takes 18 seconds/query. What two levers reduce that?

→ (1) It recovers the returns and dispute docs; the sets overlap because every variant is the same question reworded, so they mostly hit the same docs (plus a few extras) → recall. (2) The question is two intents (price + support) living in different docs; the sub-question sets are disjoint, so you need both → coverage. (3a) Neither — simple, specific; just retrieve. (3b) Multi-query — one intent, many phrasings. (3c) Decomposition (parallel) — distinct facets (uptime, price) across docs. (3d) Sequential decomposition (multi-hop) — find the acquirer first, then its CEO. (4) No — diminishing returns after 3–4; 10 variants mostly add latency, cost, and duplicates. (5) Cache decomposed sub-queries (skip the planning LLM call on repeats) and route so only genuinely-compound queries get decomposed at all.

Mental-Model Corrections

"Multi-query and decomposition are the same thing." Opposite problems: multi-query = same question, many phrasings (→ recall); decomposition = one compound question split into different sub-questions (→ coverage).
"Multi-query result sets should be disjoint." They overlap — that's the point; the union beats any single phrasing, and RRF rewards the consensus.
"Decomposition is just rephrasing." No — its sub-questions are distinct information needs retrieving disjoint docs; you need all the answers.
"More query variants = better." Diminishing returns after ~3–4; beyond that it's latency and duplicates.
"Decompose every query to be thorough." Decomposition is expensive (seconds/query) and over-splits simple questions, adding failure modes. Route — most queries need neither technique.
"Sequential decomposition is strictly better (more reasoning)." It enables multi-hop but suffers error cascades; prefer parallel when sub-questions are independent.

Key Takeaways

Multi-Query (fan-out): rephrase the same question N ways (≈3), retrieve each, fuse with RRF. Result sets overlap; the union boosts recall and handles vocabulary mismatch/ambiguity. (MultiQueryRetriever.)
Decomposition (split): break one compound/multi-hop question into different sub-questions, answer each, compose. Sub-questions retrieve disjoint docs → coverage. (SubQuestionQueryEngine.)
The distinction to memorize: overlap → recall (multi-query) vs disjoint → coverage (decomposition). Both emit multiple ranked lists → RRF fuses.
Decomposition is parallel (independent, error-isolated) or sequential/multi-hop (depends on prior answers — powerful but error cascades, the path to agentic RAG).
Honest: both multiply retrieval + LLM cost. Multi-query: 3–4 variant ceiling. Decomposition: expensive (seconds/query) — cache sub-queries, don't over-split.
Route: send each query to the minimally sufficient strategy — most need neither. That adaptive routing is next.