Retrieve → Augment → Generate (Build From Scratch)

Introduction

You know why RAG works, you've seen the whole map, and you've built the offline ingestion box. Now we build the online query path — the part that runs on every user request — from scratch, no framework. By the end you'll have a complete rag() function you wrote yourself, every line of which you understand.

Why by hand? Because a framework can give you index.as_query_engine() in three lines — and then, when retrieval is bad, you have no idea which box is broken. Production RAG is debugging, and you can only debug what you understand. We'll write each of R → A → G deliberately, and at each step do the thing well, not just at all.

In this lesson you'll learn to:

Retrieve well — embed the query, search top-k, and choose k
Augment well — build a grounded, citable prompt, and beat "lost in the middle"
Generate well — low temperature, citations, and honest refusal
Assemble the whole thing — and know when a framework finally earns its place

The Plan: R → A → G, Done Well

Here's the entire query pipeline with the real decision at each step — the difference between a demo and something you'd ship. We'll implement each box in turn, then snap them together into one rag() function. Keep this picture in mind as we go:

An infographic titled 'The Query Pipeline, Done Well — R, A, G' showing the three online RAG steps as stacked cards. R RETRIEVE (find the right chunks): embed the query with the same model and a query prefix, then vector search top-k plus a metadata filter; a note says choose k — too few misses, too many adds noise and cost, start around five. A AUGMENT (build the prompt, the make-or-break step): prompt equals instructions plus numbered, source-labelled chunks plus the question. It highlights three decisions: Order — 'lost in the middle', shown as a small U-shaped accuracy curve that is high at the start and end and sags in the middle, so put your top chunks at both ends; Budget — more context is not better, irrelevant chunks distract and cost tokens; and Grounding — answer only from context, cite sources, and say I don't know. G GENERATE (grounded answer): call the LLM, use low temperature for factual output, cite the chunks used, refuse when context is thin, and optionally return structured answer-plus-sources. A banner reads: you write every line of R to A to G, which is exactly why you can debug every box; frameworks come after you understand the pieces.

R — Retrieve Well

Retrieval turns the question into the right handful of chunks. Three parts:

Embed the query with the same model used at indexing (non-negotiable — query and passage vectors must share a space). Add the model's query: prefix if it expects one.
Vector search → top-k by cosine similarity.
Filter by metadata before (or alongside) the search — tenant, language, recency, source — using the metadata you captured at ingestion.

import numpy as np
from openai import OpenAI
client = OpenAI()
EMBED_MODEL = "text-embedding-3-large"

def embed(texts, kind="passage"):
    # Some models want a task prefix ("query:" / "passage:"); pass it through if so.
    r = client.embeddings.create(model=EMBED_MODEL, input=texts)
    v = np.array([d.embedding for d in r.data])
    return v / np.linalg.norm(v, axis=1, keepdims=True)     # unit vectors → dot == cosine

# From ingestion (L86): chunks = [{"text":..., "source":..., "page":...}, ...]
# Index built once, offline:  index = embed([c["text"] for c in chunks])

def retrieve(query, k=5, where=None):
    # metadata FILTER first (e.g. where=lambda c: c["source"] == "handbook.pdf")
    cand = [i for i, c in enumerate(chunks) if where is None or where(c)]
    q = embed([query], kind="query")[0]
    sims = index[cand] @ q                                   # cosine similarity
    top = [cand[i] for i in np.argsort(-sims)[:k]]           # top-k
    return [chunks[i] for i in top]

The one real knob here is k — and it's a genuine tradeoff:

Too small (k=1–2) → you miss the chunk that holds the answer. Fatal.
Too large (k=20) → you pull in irrelevant chunks that add noise, cost tokens, and (next step) actively distract the model.

Start around k=5 and tune on your data. (This is honest, baseline retrieval. Two upgrades — hybrid search to catch exact keywords, and a reranker to reorder the top-k by true relevance — get their own section later. They make this box better; they don't change its shape.)

A — Augment Well: The Make-or-Break Step

This is where most RAG quality is won or lost. Augment = assemble the prompt from instructions + retrieved chunks + the question. It's pure context engineering, and three decisions matter:

① Format chunks so the model can cite them. Number each chunk and label its source. Now the model can write "...within 5 business days [1]" and you can trace every claim back to a document + page.

② Ground it — hard. A plain LLM will happily answer from its memory and make things up. The system instruction must force: answer ONLY from the context, cite the chunks used, and say "I don't know" when the answer isn't there. Models do not refuse on their own — you must tell them to.

SYSTEM = (
    "You are a support assistant. Answer the question using ONLY the context below. "
    "Cite the chunk number(s) you used, like [1] or [2]. "
    "If the answer is not in the context, reply exactly: "
    "\"I don't know based on the provided documents.\""
)

def build_prompt(query, hits):
    # Number each chunk AND label its source → the model can cite, you can trace.
    blocks = [
        f"[{i+1}] (source: {h['source']}, p.{h['page']})\n{h['text']}"
        for i, h in enumerate(hits)
    ]
    context = "\n\n".join(blocks)
    return f"{SYSTEM}\n\n# Context\n{context}\n\n# Question\n{query}"

③ Mind the order — the "lost in the middle" effect. This one is deeply counterintuitive, so explore it below. Even long-context models read the beginning and end of the context far better than the middle — a U-shaped accuracy curve. Liu et al. (2024) measured a 30%+ accuracy drop when the answer document moved from position 1 to position 10 of 20. The fix isn't "most relevant first" — it's place your best chunks at both ends and let weaker ones sit in the middle, working with the model's attention bias instead of against it. (With k≈5 this is minor; it becomes critical as k grows or context gets long.)

Interactive: a lost-in-the-middle placement lab. The user retrieves k chunks (one holds the answer) and drags the answer chunk's position within the prompt while tuning k, watching predicted accuracy follow the lost-in-the-middle valley — high when the answer sits at position 1 or k (the start or end), cratering when it is buried in the middle — and dropping further as large k adds distractor chunks that dilute the signal and cost tokens. A live prompt stack shows the system instruction, the numbered chunk slots with the answer chunk highlighted at its position, and the user question, while an accuracy meter and verdict explain each state (buried in the middle, k too large, or correctly placed at an end). The two make-or-break augment levers made tangible: order (put your best chunks at both ends, not in the middle) and k (precision beats volume — start around 5). Illustrative accuracy model.

The effect is a property of how models process position, not of the content — it persists even when you shuffle the documents. Newer models have softened it, but none have eliminated it. Reordering your retrieved chunks costs nothing and is one of the cheapest wins in RAG.

G — Generate Well

Generation is the easiest box to call and the easiest to do carelessly. The LLM writes the answer from the provided context — so set it up to stay grounded:

Low temperature (0 or near-0). This is factual Q&A, not creative writing — you want faithful, low-drift answers, not flair.
Citations — already instructed in the system prompt; verify the model actually includes them.
Honest refusal — when the context doesn't contain the answer, the model should say so (you told it to). This is what makes RAG trustworthy.
(Optional) structured output — return {answer, citations} as JSON when a downstream system needs to parse it.

def generate(prompt):
    out = client.chat.completions.create(
        model="gpt-5.5",
        temperature=0,                       # factual grounding → deterministic, low drift
        messages=[{"role": "user", "content": prompt}],
    )
    return out.choices[0].message.content

Notice how little code this is. That's the 2026 reality from the architecture lesson: the LLM call is rarely the bottleneck — retrieval is. Get the right chunks in, framed well, and a modern model handles the rest.

Put It Together: The Whole `rag()`

Three small functions, one pipeline. Here's the complete, from-scratch query path — retrieve → augment → generate — returning a grounded answer and its sources:

def rag(question, k=5, where=None):
    hits   = retrieve(question, k=k, where=where)   # R — find the right chunks
    prompt = build_prompt(question, hits)           # A — assemble grounded prompt
    answer = generate(prompt)                       # G — LLM answers from context
    sources = [{"source": h["source"], "page": h["page"]} for h in hits]
    return {"answer": answer, "sources": sources}

# In the corpus → grounded, cited answer:
print(rag("how long do refunds take?"))
# {'answer': 'Refunds are processed within 5 business days of approval. [1]',
#  'sources': [{'source': 'handbook.pdf', 'page': 12}, ...]}

# NOT in the corpus → honest refusal, no hallucination:
print(rag("what is your dividend policy?"))
# {'answer': "I don't know based on the provided documents.", 'sources': [...]}

That's a real RAG system: metadata-filtered retrieval, a grounded + citable prompt, low-temperature generation, source tracking, and honest refusal when the answer isn't in your documents. Swap the numpy index for a real vector DB and add reranking, and the shape never changes — you just made each box better.

Build From Scratch — Then Reach for a Framework

Could LangChain or LlamaIndex have done this in a few lines? Absolutely — index.as_query_engine().query(...). So why hand-roll it?

Because you just made every box visible and debuggable. When your shipped RAG returns a wrong answer, you now know exactly where to look: Did retrieve return the right chunk? (R) Was it framed and ordered well? (A) Did the model ground its answer? (G) Framework users who skipped this step are stuck poking a black box.

The honest, balanced take:

Frameworks are genuinely useful — loaders, integrations, retrievers, reranker hooks, eval tools. Use them.
But adopt them after you understand the pieces, not instead of. The danger isn't the framework; it's reaching for it before the mental model, so you can't tell what it's hiding or fix it when it breaks.

Build it once by hand — then use whatever you like, with your eyes open. That's the difference between someone who uses RAG and someone who can ship and maintain it.

🧪 Try It Yourself

Drive the interactive, then debug the pipeline:

In the widget: drag the answer chunk from position 1 → 10 → 20. Where is accuracy highest? Lowest? If a reranker hands you 8 chunks ordered best→worst, in what order should you actually place them in the prompt?
Localize three bugs in a rag() you built — name the box (R, A, or G) and the fix:
(a) The right document is clearly in your corpus, but the answer says "I don't know."
(b) The answer is fluent and confident but cites a fact that's nowhere in your documents.
(c) The answers change slightly every time you ask the same question.

→ (1) Highest at the ends (positions 1 and 20), lowest in the middle (~10). Place the reranked chunks so the best sit at the start and end, weakest in the middle. (2a) R — retrieve didn't surface it: bump k, check the metadata where filter, or you need hybrid/reranking (later). (2b) A or G — weak grounding: tighten the "answer ONLY from context" instruction and confirm the chunk was even retrieved (could also be R). (2c) G — temperature isn't 0; set it for factual grounding. Notice: naming the box is the whole skill.

Mental-Model Corrections

"RAG is one magic call." It's three boxes you control — R, A, G — each with a real decision. Building them by hand is what makes RAG debuggable.
"More retrieved chunks = better answers." No — past a point, extra chunks distract the model and cost tokens. Tune k (~5 to start); precision beats volume.
"Order doesn't matter — the model reads everything." Lost in the middle: models favor the start and end. Put your best chunks at both ends.
"The model will refuse if it doesn't know." It won't — by default it answers from memory and makes things up. You must instruct grounding, citations, and "I don't know."
"Use a framework from day one." Build it from scratch first so you understand every box; then adopt a framework with your eyes open.
"Generation is the hard part." It's the small part — retrieval is the bottleneck. Good context in → good answer out.

Key Takeaways

The online query path is R → A → G, and you can build it from scratch in ~40 lines: retrieve → build_prompt → generate.
Retrieve: same embedding model + query: prefix, top-k vector search, metadata filter; tune k (too few misses, too many adds noise — start ~5).
Augment (make-or-break): number + source-label chunks (for citations), ground hard ("answer only from context · cite · say I don't know"), and order for "lost in the middle" — best chunks at both ends.
Generate: low temperature, real citations, honest refusal; the LLM call is the small part — retrieval is the bottleneck.
Build it by hand, then use a framework — so every box stays visible and debuggable. Next: citing sources & grounding answers, turning that [1] into trustworthy, verifiable references.