A Production Retrieval Stack (Putting It Together)

Introduction

You've now learned every major advanced-retrieval technique: hybrid search, RRF fusion, reranking, query rewriting, HyDE, multi-query, decomposition, and routing. This capstone answers the question that actually matters in production: how do they fit together — and which ones should you actually use?

Because here's the trap. With this many powerful tools, the temptation is to bolt all of them onto every query. That path leads to a system that's slow, expensive, fragile, and often worse than a clean baseline. The mark of a strong AI engineer isn't using every technique — it's assembling the right stack and knowing what to leave out.

In this lesson:

The two pipelines (offline indexing + online query) and the canonical stage order
Why you start simple and add complexity only to fix a measured failure
Evaluation-driven development — each stage must earn its place
The cost–latency–quality budget, caching, and observability

Two Pipelines: Offline & Online

Every production RAG system is really two systems with very different constraints — keep them separate in your head:

⚙️ The offline indexing pipeline runs as an idempotent batch job, triggered by document changes (and a full re-index scheduled periodically to absorb embedding-model or chunking upgrades). It's throughput-bound, not latency-bound — it can take its time. Stages (all from this container): ingest → clean/parse → chunk → enrich (metadata, parent-child) → embed → store.

⚡ The online query pipeline runs as a low-latency service with explicit cost and latency budgets per stage — users are waiting. This is where everything from the Advanced Retrieval section lives, and getting its order right is the core of this lesson.

The architectural principle to remember from the whole container: in 2026, retrieval is the bottleneck, not generation. Modern LLMs synthesize beautifully from correct context — so almost all of your engineering effort belongs in this pipeline, getting the right context to the model.

An infographic titled 'The Production Retrieval Stack — Put It Together'. A dashed OFFLINE strip (indexing, idempotent batch, re-index for model or chunking upgrades) shows ingest, clean, chunk, enrich (metadata / parent-child), embed, store. Below, the ONLINE query pipeline (with per-stage cost and latency budgets) is a left-to-right chain of stages, each tagged with its lesson: query, then route (L101), then transform (L99-100), then hybrid retrieve plus metadata filter (L96, L81), then RRF fuse (L97), then rerank (L98), then augment plus cite (L87-88); an axis underneath reads left = cheap, wide net (recall) and right = expensive, narrow sort (precision). Two cards: one, Add a stage to fix a NAMED failure — recall miss means hybrid plus query transform, precision miss means rerank, multi-hop miss means decompose or graph; order hybrid, then rerank, then transforms, then graph; each stage must earn its latency. Two, Evaluation-driven plus CLQ budget — baseline, then measure (recall@k, nDCG, MRR, faithfulness), then add only what moves the metric; balance Cost-Latency-Quality (e.g. rerank plus 3 percent but plus 40 percent dollars?); cache plus route; observe in production. Banner: Hybrid plus RRF plus rerank is the near-SOTA 2026 default; don't build the whole stack on day one — measure, and let each stage earn its place.

The Canonical Order: Recall, Then Precision

Here is the full online pipeline, assembled — every stage tagged with the lesson it came from. Read it as one story:

def rag_pipeline(query, history=None):
    # 0. ROUTE (L101): what does THIS query actually need?
    plan = router.route(query)                       # {needs_retrieval, fan_out, source, filter}
    if not plan.needs_retrieval:
        return llm(query)                            # chitchat / known → skip retrieval entirely

    # 1. TRANSFORM (L99–L100): reshape the query BEFORE searching
    q = rewrite_standalone(query, history)           # conversational follow-up → standalone
    queries = multi_query(q) if plan.fan_out else [q]  # fan out / decompose if the router said so

    # 2. RETRIEVE (L96 + L81): hybrid (dense + BM25) + metadata filter — a wide, cheap net
    lists = [hybrid_search(qi, where=plan.filter, k=50) for qi in queries]

    # 3. FUSE (L97): reciprocal rank fusion across every list (scale-free, consensus-aware)
    candidates = reciprocal_rank_fusion(lists, k=60)

    # 4. RERANK (L98): a cross-encoder reorders by true relevance; keep only the best few
    top = reranker.rerank(query, candidates)[:5]

    # 5. AUGMENT + GENERATE + CITE (L87–L88): grounded answer with sources
    return generate_with_citations(query, top)

The order is not arbitrary — it follows one governing principle you've seen since reranking: a cheap, wide net for recall, then expensive, precise sorting on the survivors.

Route first (cheapest decision) — skip everything you don't need.
Transform the query — fix it before it touches the index.
Hybrid retrieve + filter — cast the wide net over millions, cheaply (dense for meaning, BM25 for exact terms).
RRF fuse — merge the lists scale-free (≈free).
Rerank — now run the expensive cross-encoder, but only on ~50–100 candidates (the "priciest model on the fewest items" rule).
Augment + generate + cite — grounded answer with sources.

Each step narrows and sharpens: lots of cheap candidates → a few precise ones → one grounded answer. Put an expensive stage too early (e.g. rerank the whole corpus) and it's impossibly slow; put recall too late and you can't recover what you never retrieved. This order is the production stack.

See It: Build the Stack

Before you build it for real, feel the tradeoffs. Toggle stages below and watch two bars move in tension: retrieval quality (rising, with diminishing returns) and average latency (rising — some stages cheap, some brutal):

Toggle stages and watch retrieval quality climb (with diminishing returns) while latency rises. Hybrid + RRF + rerank is the sweet spot; add an LLM transform and latency explodes — enable routing and watch it drop back.

Three lessons jump out of the widget:

Hybrid + RRF + rerank is the sweet spot — big quality jump for modest latency. This is the near-SOTA 2026 default for most apps.
Per-query LLM transforms have a steep price — ~+500ms for a few quality points. A classic "is it worth it?" decision.
Routing is the only toggle that can lower average latency — by running the expensive transform only on the queries that need it. That's why routing (L101) is what makes a heavy stack affordable.

Start Simple — Then Add to Fix a Named Failure

Do not build the full pipeline on day one. The professional path is to ship a baseline, measure it, and add complexity surgically. Your baseline is genuinely good:

# Day 1 — ship THIS, measure it, and only then add stages.
def baseline_rag(query):
    chunks = retriever.search(embed(query), k=5)     # recursive chunks + a strong embedding model
    return generate_with_citations(query, chunks)
# A good baseline answers most queries. Every stage you add must beat it ON YOUR EVAL.

Recursive chunking + a strong embedding model + top-k answers a surprising fraction of queries. Ship it, then let measured failures drive what you add — because each failure mode has a specific cure:

The failure you measure	The stage that fixes it
Recall — the right doc isn't retrieved	Hybrid search + query transformation
Precision — right doc retrieved but ranked low / noise on top	Reranking
Multi-hop — needs facts across several docs	Decomposition / GraphRAG
Vague/conversational queries fail	Query rewriting

The recommended build order falls right out of this: hybrid search → reranking → query transforms → (graph for multi-hop). And the discipline from the last three lessons crystallizes into one rule:

Never add a stage "to be thorough." Add it only when your eval shows the specific failure it cures — and only if it earns its latency. Complexity you can't justify with a metric is expensive theatre: more cost, more latency, more failure modes, no benefit.

Evaluation-Driven Development

That rule only works if you can measure, which is why the single most important habit in production RAG is evaluation-driven development (EDD) — the RAG analogue of test-driven development. Component selection is evaluation-driven, not preference-based.

The practice:

Build an eval set that mirrors your real query distribution — common questions, hard multi-doc synthesis questions, and edge cases. A few dozen labeled (query → relevant docs / good answer) examples is enough to start and worth its weight in gold.
Measure two layers separately:
- Retrieval: recall@k (are the right docs in the top-k?), nDCG / MRR (are they ranked high?). Retrieval quality caps everything downstream.
- End-to-end: faithfulness (is the answer grounded in the context?) and answer relevance — the RAGAS metrics you'll meet in depth in L107.
Change one thing, re-measure. Add hybrid → did recall@k move? Add rerank → did nDCG move? Keep the change only if the metric improved enough to justify its cost.

This flips RAG from vibes-based tinkering ("this prompt feels better") into engineering: every stage in your stack is there because a number says it earns its place. It's also the only way to safely upgrade later (new embedding model? re-run the eval) and to catch regressions in CI.

The Cost–Latency–Quality Budget

Production retrieval is a constrained optimization: you're trading Cost, Latency, and Quality (CLQ) against each other, and you rarely get all three. Treat your pre-generation choices — embedder, ANN settings, candidate depth k, hybridization, reranking, transforms — as dials on that budget.

Latency. Aim for sensible targets (e.g. sub-300ms P50, under ~2s P95, streaming the answer so time-to-first-token stays low). Every stage spends from this budget — which is exactly what the interactive showed.

Cost. In a typical system, embedding generation (40–60%) and vector storage (20–35%) dominate — often more than LLM inference. The big levers:

Caching — cache query embeddings, query rewrites, and decomposed sub-queries; semantic caching of whole answers for repeated questions. (A cost-control layer of caching + routing + token budgeting can cut LLM spend ~80%+.)
Routing (L101) — the system-wide version of "spend proportional to difficulty."
Candidate depth k — bigger k raises recall and rerank cost; tune it.

The mindset, and the reason observability matters: instrument the pipeline so you can correlate each dial with quality and spend — e.g. "reranking improves nDCG by 3% but costs 40% more" is a decision you should be able to see in a dashboard, not guess. Log queries, retrieved docs, scores, routes, and cache-hit rates; monitor recall/nDCG and latency in production; watch for drift; gate changes behind your eval in CI. A stage that wins on your offline eval but blows the latency budget doesn't ship.

🧪 Try It Yourself

Use the stack builder, then make production calls:

Starting from baseline, enable Hybrid, then RRF, then Rerank. Where's the biggest quality jump per millisecond? Where do returns start diminishing?
Enable the Query transform (no routing). What happens to latency vs quality? Now enable Routing — explain why average latency drops.
Order these stages as you'd add them to a baseline, and name the failure each fixes: reranking, GraphRAG, hybrid search, query rewriting.
Your eval shows reranking lifts nDCG +3% but adds +40% cost and 130ms. Ship it or not — and what determines the answer?
A teammate proposes shipping hybrid + multi-query + decomposition + HyDE + rerank from day one for a brand-new internal docs bot. What's your counsel?

→ (1) Hybrid and especially rerank give the biggest quality-per-ms; stacking LLM transforms on top is where returns diminish (small quality, large latency). (2) Transform adds ~500ms to every query for a few points; routing runs it only on the ~third of queries that need it, so the average latency falls (most queries skip the costly step). (3) Hybrid search (recall) → reranking (precision) → query rewriting (vague/conversational) → GraphRAG (multi-hop) — the failure-mode order. (4) It depends on the budget and the use case: ship it if 3% nDCG matters for this product and the latency/cost fit the budget; skip it for a cost-sensitive, latency-critical app where 3% won't be felt. The point is you can only decide because you measured both sides. (5) Push back: start with a baseline + maybe hybrid, measure, and add stages only to fix failures the eval reveals. Shipping the whole stack blind is expensive theatre — slow, costly, hard to debug, and likely no better than hybrid+rerank.

Mental-Model Corrections

"More retrieval techniques = better RAG." No — they add latency, cost, and failure modes. Add only what a measured failure demands; the best stack is often small.
"Stage order doesn't matter." It's the core design: recall (cheap, wide) → precision (expensive, narrow). Route → transform → hybrid retrieve → fuse → rerank → generate.
"Indexing and querying are one pipeline." Two systems: offline batch (throughput-bound, re-indexable) vs online service (latency-bound, budgeted).
"Tune RAG by intuition." Tune it by evaluation — recall@k / nDCG / MRR and faithfulness on an eval set. Evaluation-driven, not preference-driven.
"Optimize for quality alone." Production is Cost–Latency–Quality; a stage that wins offline but blows the latency budget doesn't ship. Cache and route to afford quality.
"Build the full stack on day one." Ship a baseline, measure, add surgically. Most apps land at hybrid + RRF + rerank.

Key Takeaways

Production RAG = two pipelines: an offline idempotent indexing batch (ingest→chunk→enrich→embed→store) and an online latency-budgeted query service.
The canonical online order is recall → precision: route (L101) → transform (L99–100) → hybrid retrieve + filter (L96/L81) → RRF fuse (L97) → rerank (L98) → augment + cite (L87–88). Cheap wide net first; expensive precise sort last.
Start with a baseline; add a stage only to fix a named, measured failure: recall→hybrid+transform, precision→rerank, multi-hop→decompose/graph. Order: hybrid → rerank → transforms → graph.
Evaluation-driven development: an eval set + recall@k / nDCG / MRR (retrieval) and faithfulness (end-to-end); each stage must earn its place on the metric.
Cost–Latency–Quality budget: dials = embedder, k, hybrid, rerank, transforms; cache + route to afford quality; observe in prod (latency, nDCG, cache hits, drift) and gate changes in CI.
The near-SOTA 2026 default is hybrid + RRF + rerank — don't build more until your eval says you need it. That completes Advanced Retrieval. Next container: GraphRAG, multimodal, and agentic RAG — for the multi-hop and complex cases this stack still can't reach.