Adaptive Retrieval Routing

Introduction

Over the last five lessons you built a powerful retrieval toolbox — hybrid search, RRF fusion, reranking, query rewriting, HyDE, multi-query, decomposition. But there's a trap hiding in all that power: if you run every technique on every query, you've built something slow, expensive, and often worse. A simple "what's the refund window?" doesn't need a 5-query fan-out and a decomposition planner — and a "thanks!" doesn't need retrieval at all.

Adaptive retrieval routing is the answer, and it's the lesson that ties this whole section together. A router sits at the front door of your RAG system, inspects each incoming query, and decides — per query — what to actually do: retrieve or not, how much machinery, and from which source. The goal: spend compute proportional to difficulty.

In this lesson you'll learn:

The two routing dimensions — how much (depth) and from where (source)
How routers decide — semantic, LLM, and trained classifiers (and Self-RAG)
The big payoff: matching always-expensive accuracy at a fraction of the cost
The single-point-of-failure risk — and the bridge to agentic RAG

The Router: Your System's Front Door

Naive RAG has one fixed pipeline that every query marches through. Routing replaces that with a decision layer: a small, fast component that reads the query and dispatches it. Think of it as a triage nurse or a switchboard — look at what came in, send it to the right place with the right urgency.

The router classifies the query and picks a route — and crucially, most queries take a cheap route:

# An LLM router: classify the query, then dispatch to the minimally-sufficient strategy.
ROUTER_PROMPT = """Classify the user query and choose ONE route. Reply as JSON
{"route": "...", "confidence": 0.0-1.0}.

Routes:
- none      : chitchat or something you can answer directly (no lookup needed)
- direct    : a single, specific fact → one retrieval
- multi     : vague/ambiguous wording → multi-query fan-out
- decompose : compound or multi-hop question → split into sub-questions
- sql       : an aggregate/number over structured data → query the database
- web       : fresh or external info → web search

Query: {q}"""

def route(query):
    decision = json.loads(llm(ROUTER_PROMPT.format(q=query)))
    if decision["confidence"] < 0.6:                 # ← don't act on a weak guess
        return ask_clarifying_question(query)        #    surface uncertainty, then re-route
    return DISPATCH[decision["route"]](query)        # run only the chosen pipeline

# Most queries hit 'none'/'direct' (cheap). Only the genuinely hard ones pay for
# 'decompose'. Cost & latency end up PROPORTIONAL to difficulty.

An infographic titled 'Adaptive Retrieval Routing — Spend Compute Proportional to Difficulty'. It notes the earlier lessons built a powerful but expensive toolbox, so running it all on every query is wasteful; the router is the system's front door that decides, per query, what to actually do. A fan-out diagram: a query goes into a ROUTER (classify intent plus difficulty) which branches to six routes — no retrieval, direct (single step), multi-query, decompose, SQL/DB, and web search. Two axis cards: One, How MUCH (depth) — Adaptive-RAG routes by complexity into A no-retrieval, B single-step, C multi-step, so retrieval depth is proportional to difficulty and matches always-expensive accuracy at far lower cost. Two, From WHERE (source) — send the query to the right source: doc index, a SQL database for aggregates and numbers, web search for fresh info, or a specific knowledge base or tool, via RouterQueryEngine. A strip on how the router decides: semantic (embed query vs route examples, fast, no LLM), LLM (function-calling picks a tool, flexible), trained classifier (small model like Adaptive-RAG's T5, fastest), or Self-RAG where the model itself decides to retrieve on demand. A caveat: the router is a single point of failure — a misroute means no retrieval when it was needed (causing hallucination) or the wrong source; the number-one reliability lever is a confidence threshold plus a clarifying-question fallback (don't act on a weak guess) and a cap on hops so it can't loop; a router that loops and decides iteratively IS an agent, the bridge to agentic RAG. Banner: decide per query — retrieve or not, how much, from where — and spend compute proportional to difficulty; the front door of every serious RAG system.

Now a greeting costs ~nothing, a simple lookup does one retrieval, and only a genuinely compound question pays for decomposition. Cost and latency become proportional to difficulty instead of pinned at the maximum for everything. (Notice the confidence < 0.6 check — we'll come back to why that line is the most important one in the function.)

Two Things a Router Decides: How Much, and From Where

Routing operates on two independent axes — keep them distinct:

① How MUCH? — depth routing. This is the "spend proportional to difficulty" axis, and the canonical method is Adaptive-RAG (Jeong et al., 2024). A lightweight classifier sorts each query into three complexity levels:

A — No retrieval: the model already knows it (or it's chitchat). Answer directly.
B — Single-step retrieval: one lookup suffices.
C — Multi-step retrieval: iterative / multi-hop retrieval (decomposition).

The result is the headline finding: a three-class complexity router matches the accuracy of always doing expensive multi-step retrieval — at substantially lower cost and latency, because most queries don't need the heavy path.

② From WHERE? — source routing. A query should go to the right knowledge source, which often isn't your vector index:

"What were total sales in Q3?" → a SQL database (aggregates over structured data — vector search is terrible at counting).
"What's our competitor's latest pricing?" → web search (fresh, external — your static index can't have it).
"What's our PTO policy?" vs "How do I deploy?" → different indexes (HR vs engineering docs).

LlamaIndex's RouterQueryEngine is the canonical tool: it picks which query engine (index, DB, tool) handles each query — single-select (one route) or multi-select (fan to several, then aggregate). Often a real router does both axes — classify the source and the depth.

See It: Route Each Query to Its Cheapest Sufficient Path

This widget is the lesson. Click through queries spanning the spectrum — from "thanks!" to a compound pricing-and-support question to an aggregate that belongs in SQL — and watch the router choose a route, explain why, and pay only that route's cost. Compare it to the cost of blindly running the full pipeline every time:

Pick a query and watch the router choose a route (no-retrieval → decompose, or SQL/web) with its rationale and cost — vs the cost of always running the full pipeline. Notice cost scales with difficulty.

The summary line is the whole argument: across a realistic mix, adaptive routing costs a fraction of always-on, and the easy queries are also faster. You're not sacrificing quality on the hard queries — they still get decomposition and reranking — you're just not wasting that machinery on the easy ones. Spend proportional to difficulty.

How the Router Actually Decides

There are three ways to build the decision-maker, trading cost, flexibility, and setup:

1. Semantic router (embedding-based). Embed a few example utterances per route once; at query time, embed the query and pick the nearest route by cosine similarity. No LLM call → microsecond, ~free decisions. Best for stable, well-separated routes (e.g. topic → index).

# A SEMANTIC router — no LLM call at decision time. Embed route examples once;
# route each query to the nearest set (then run that route's pipeline).
ROUTES = {
    "billing":  ["refund", "invoice", "payment failed", "charge dispute"],
    "account":  ["reset password", "2FA", "can't log in", "change email"],
    "sql":      ["how many", "total sales", "average", "count of"],
}
route_centroids = {name: embed(examples).mean(0) for name, examples in ROUTES.items()}

def semantic_route(query):
    q = embed([query])[0]
    return max(route_centroids, key=lambda r: cosine(q, route_centroids[r]))
# Microsecond decisions, no token cost — great for stable, well-separated routes.

2. LLM router. Prompt an LLM to select a route or tool (LlamaIndex's LLMSingleSelector emits JSON; the Pydantic/function-calling variant uses the tool-call API). Flexible and zero-training — it can reason about nuanced or novel queries — but it adds an LLM call of latency and cost to every request.

3. Trained classifier. A small fine-tuned model predicts the route. Adaptive-RAG trains a T5-Large on automatically-derived complexity labels (run each query through no-/single-/multi-step pipelines and label by which first succeeds). Fastest at inference and cheap to run, but needs training data. (Even KNN/MLP on sentence embeddings are competitive — and lightweight routers can halve API costs by sending easy queries to a weak model and hard ones to a strong model.)

And there's a fourth, more radical option: let the model route itself. Self-RAG fine-tunes an LLM to emit reflection tokens ([Retrieve], [ISREL], [ISSUP], [ISUSE]) that decide when to retrieve on demand, mid-generation, and judge the results — no separate router at all. Self-Routing RAG generalizes this: the model decides whether to retrieve externally or just answer from its own parametric knowledge. This is where routing starts to merge with the model itself.

The Honest Take: The Router Is a Single Point of Failure

Routing is powerful, but you've just put a decision in front of your whole pipeline — and that decision can be wrong. This is the risk to respect:

A misroute is often worse than no routing at all:

Route to none when retrieval was needed → the model hallucinates confidently from memory.
Route to the wrong source → a correct-looking answer from the wrong data.
Route too shallow on a multi-hop question → a confident, incomplete answer.

The router is a single point of failure, and every routing hop is "another place a decision can go wrong, plus a few ms of latency." So:

The #1 reliability lever is a confidence threshold with a clarifying-question fallback. When the router's confidence is low, don't act on a weak guess — ask one targeted question (or fall back to a safe default route) and re-route once intent is clear. A router that surfaces its low-confidence decisions is far more reliable than one that hides them behind a confident-looking but wrong dispatch. (That confidence < 0.6 line from earlier — that was it.)

Two more guardrails: monitor route accuracy (it's a classifier — measure it, and watch for drift), and if routes can hand off to each other, enforce a hard hop limit so a chain can never loop forever.

And don't over-engineer the router itself. A simple 2–3 route LLM or semantic classifier covers most apps; reach for a trained complexity classifier (Adaptive-RAG style) at scale, where the per-query LLM-router cost adds up.

Routing Is the Doorway to Agentic RAG

Here's the idea that makes routing feel bigger than it looks: a router is the simplest possible agent. It's an LLM (or model) making a decision about what action to take based on the input. Once you let that decision-maker loop — retrieve, look at what came back, decide whether it's enough, and if not, reformulate and retrieve again — you no longer have a static pipeline. You have an agent.

That's not hypothetical: Self-RAG already decides during generation whether to retrieve and whether the results are good enough; Self-Route uses the LLM's own calibration to decide if a query is answerable. The line between "a router that can re-route" and "an agentic RAG system" is genuinely blurry — routing is where retrieval stops being a fixed function and starts being a decision process.

That's exactly where the next container goes. You now have a complete advanced retrieval toolkit and the routing layer that applies it intelligently. The final step is letting the system reason about retrieval in a loop — corrective and self-reflective RAG — which is the heart of agentic RAG.

🧪 Try It Yourself

Drive the router widget, then design like an architect:

Click "thanks, that helps!" and the compound pricing-and-support question. How different are their routes and costs? What would running the full pipeline on the greeting have wasted?
"What were total sales in Q3 2024?" routes to SQL, not the vector index. Why would semantic search over documents be a bad choice for this query?
Your router classifies an ambiguous query with confidence 0.45. What should the system do — and why is that more reliable than just picking the top route?
Pick the router type (semantic / LLM / trained) for: (a) a fixed set of 4 well-separated product areas; (b) a research assistant facing wildly varied, novel queries; (c) a high-traffic product where per-query LLM cost matters and you have labeled data.
Your routed system sometimes confidently answers from memory and is wrong. Which misroute is happening, and what's the fix?

→ (1) The greeting → no-retrieval (~1 unit); the compound question → decompose (~8). Full-pipeline on the greeting wastes a fan-out + decomposition + rerank it never needed — pure cost/latency for nothing. (2) Aggregates/counts require exact computation over structured rows; embeddings approximate meaning, not arithmetic — vector search can't reliably sum sales. Route to SQL. (3) Don't act on the weak guess — ask a clarifying question (or use a safe default) and re-route; a confident wrong dispatch is worse than admitting uncertainty. (4a) Semantic (stable, separated, no LLM cost); (4b) LLM (flexible, handles novelty); (4c) trained classifier (fast, cheap at scale, you have labels). (5) It's routing to none (no-retrieval) when retrieval was needed → it answers from parametric memory and hallucinates; raise the bar for the no-retrieval route (or add a grounding/verification check) and lean toward retrieving when unsure.

Mental-Model Corrections

"Run all my retrieval techniques on every query for best results." That's slow, costly, and often worse. Route — spend compute proportional to difficulty.
"Routing is just picking which index." That's one axis (source). The other is depth — how much retrieval (no / single / multi-step), à la Adaptive-RAG.
"Every query needs retrieval." No — chitchat and things the model knows need none. Routing (and Self-RAG) decide whether to retrieve at all.
"An LLM router is always best." It's flexible but adds a call per query. Semantic routers are ~free for stable routes; trained classifiers are fastest at scale.
"A confident route is a correct route." The router can misroute (it's a single point of failure). Use a confidence threshold + clarifying-question fallback; a wrong-but-confident dispatch is the worst case.
"Routing is unrelated to agents." A router is the simplest agent — a decision about what to do. Let it loop and you have agentic RAG.

Key Takeaways

Adaptive routing puts a decision layer at your RAG front door: per query, decide retrieve or not, how much, and from where — spending compute proportional to difficulty.
Two axes: depth (Adaptive-RAG: A no-retrieval / B single-step / C multi-step — matches always-expensive accuracy at far lower cost) and source (RouterQueryEngine: doc index / SQL / web / tool).
How it decides: semantic (embed vs route examples — fast, no LLM), LLM (function-calling — flexible), trained classifier (Adaptive-RAG's T5 — fastest at scale); or Self-RAG, where the model retrieves on demand via reflection tokens.
Honest risk: the router is a single point of failure — a misroute → hallucination or wrong source. The #1 lever is a confidence threshold + clarifying-question fallback; also monitor route accuracy and cap hops. Don't over-build the router.
The bridge: a router is the simplest agent; let it loop and re-decide and you have agentic RAG.
That completes the core advanced-retrieval techniques. Next (L102) we assemble the full production retrieval stack — and then the final section: GraphRAG, multimodal, and agentic RAG.