The End-to-End RAG Architecture

Introduction

The last lesson made the case for RAG — retrieval beats memorization: give the model the right facts at question-time instead of hoping they're baked into its weights. This lesson is the blueprint: the complete architecture of a RAG system, end to end, so you have the whole map before we build each piece.

Here's the payoff of seeing it once: every RAG system — the simplest demo and the most advanced production stack — is the same two-phase shape. Master that shape and the entire rest of this container (chunking, retrieval, reranking, GraphRAG, agentic RAG) clicks into place as upgrades to specific boxes in this one diagram.

In this lesson you'll learn:

The two phases every RAG system splits into
Each stage of the indexing and query pipelines, and what it does
The whole thing in ~25 lines so you can see the shape run
That RAG is simple to draw, deep to do well — and where the depth lives

The Big Idea: Two Phases

The single most important thing to internalize: a RAG system is two separate pipelines that meet at the vector database.

① INDEXING — offline. Build the knowledge base. You take your documents, break them up, turn them into vectors, and store them. You do this ahead of time (and then incrementally as data changes) — not while a user waits. This is your searchable memory.

② QUERYING — online. Answer a question. For each user request, you turn the question into a vector, retrieve the most relevant stored chunks, augment a prompt with them, and generate an answer grounded in that context.

The two phases share one thing — the vector database: indexing writes to it, querying reads from it. That's the whole architecture. Everything else is detail.

An infographic titled 'The RAG Architecture — Index Offline, Answer Online' showing a RAG system as two connected phases. The top band, INDEXING (offline, build the knowledge base once then incrementally), is a left-to-right pipeline: Documents (PDF, web, DB) -> Load and Clean (parse, normalize) -> Chunk (retrieval-sized) -> Embed (to vectors). In the middle sits a shared VECTOR DATABASE box holding chunk vectors plus metadata and an ANN index; the indexing phase writes down into it and the query phase reads up from it. The bottom band, QUERYING (online, answer a question every request), is another left-to-right pipeline: Query -> Embed (same model) -> Retrieve (top-k plus filter) -> Augment (prompt = context plus question) -> Generate (LLM, grounded) -> Answer (plus citations). A note reads: R-A-G = Retrieve, Augment, Generate; same embedding model both sides; in 2026 the bottleneck is retrieval, not generation, so most quality work is in chunking and retrieval, and the LLM answers only from the provided context. A banner concludes: index your knowledge offline, then retrieve, augment, generate online — that two-phase shape is every RAG system.

The Indexing Pipeline (Offline)

Walk the top row of the diagram — each box is a stage, and each maps to a part of this course:

Load & Clean — get raw text out of your sources: PDFs, HTML, Word docs, databases, APIs. Strip boilerplate, fix encoding, extract useful metadata (title, date, source, author). Garbage in → garbage retrieved. → covered next lesson (Ingestion).
Chunk — split each document into retrieval-sized passages. Too big = noisy, imprecise matches; too small = lost context. This is one of the biggest levers on RAG quality. → its own section ahead (a current default: ~300–500 tokens, 10–15% overlap, but it depends).
Embed — run each chunk through your embedding model to get a vector (Section 1). Use a passage: prefix if your model asks for one.
Store — upsert each (vector + chunk text + metadata) into the vector DB (Section 2), with deterministic IDs so re-ingesting updates rather than duplicates.

You run this once to bootstrap, then incrementally as documents are added, changed, or removed (the index lifecycle).

The Query Pipeline (Online)

Now the bottom row — what happens on every user request:

Embed the query — vectorize the question with the same embedding model used for indexing (this is non-negotiable — query and passage vectors must live in the same space). Use a query: prefix if your model expects one.
Retrieve — nearest-neighbor search in the vector DB → the top-k most similar chunks, optionally narrowed by a metadata filter (e.g. tenant_id, lang, recency). This is the R in RAG. (Advanced: add keyword/hybrid search and a reranker — a later section.)
Augment — assemble the prompt: system instructions + the retrieved chunks + the question. This is the A in RAG, and it's pure context engineering — what you put in the window, and how you frame it, decides the answer.
Generate — the LLM writes the answer grounded in the provided context (not its trained-in memory), ideally citing which chunk each claim came from, and saying "I don't know" when the context doesn't contain the answer. This is the G in RAG.

R → A → G. Retrieve, Augment, Generate. That's the name, and that's the order.

The Whole Thing in ~25 Lines

Words are abstract — here's the entire architecture as runnable code. Notice it's literally the two phases: build an index, then embed → retrieve → augment → generate. (This is a teaching skeleton — a numpy "vector DB" and no chunking — so you can see the shape; we make each box production-grade across the rest of the course.)

import numpy as np
from openai import OpenAI
client = OpenAI()

def embed(texts):
    r = client.embeddings.create(model="text-embedding-3-large", input=texts)
    v = np.array([d.embedding for d in r.data])
    return v / np.linalg.norm(v, axis=1, keepdims=True)   # unit vectors → dot = cosine

# ---------- ① INDEX (offline): load → chunk → embed → store ----------
docs = [
    "Refunds are processed within 5 business days of approval.",
    "Reset your password from the 'Forgot password?' link on the login screen.",
    "Our support team is available 9am–6pm PT, Monday through Friday.",
]
index = embed(docs)          # in production this is your vector DB (pgvector / Qdrant / …)

# ---------- ② QUERY (online): embed → retrieve → augment → generate ----------
def rag(question, k=2):
    q = embed([question])[0]
    top = np.argsort(-(index @ q))[:k]                       # RETRIEVE: top-k by cosine
    context = "\n".join(f"- {docs[i]}" for i in top)         # the retrieved chunks
    prompt = (                                                # AUGMENT: context + question
        "Answer using ONLY the context below. "
        "If the answer isn't there, say you don't know.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    out = client.chat.completions.create(                     # GENERATE: grounded answer
        model="gpt-5.5",
        messages=[{"role": "user", "content": prompt}],
    )
    return out.choices[0].message.content

print(rag("how long do refunds take?"))
# → "Refunds are processed within 5 business days of approval."  (grounded — not memorized)

That's RAG. The model answers "5 business days" not because it memorized your refund policy, but because you retrieved that fact and put it in the prompt. Swap numpy for a real vector DB, add real chunking and reranking, and you have production RAG — same shape.

Simple to Draw, Deep to Do Well

Here's the honest truth that this whole container exists to address: the diagram is simple; doing each box well is where the engineering lives. Every box hides decisions that make or break quality:

Load & Clean — did you actually extract clean text, or PDF garbage with broken tables?
Chunk — the biggest quality lever. Size, overlap, and strategy (a whole section).
Embed — model choice, dimensions, domain fit (Section 1).
Retrieve — top-k, metadata filters, hybrid search, reranking (a whole section).
Augment — how much context, in what order ("lost in the middle"), how framed.
Generate — grounding, citations, refusing when context is missing.

And a sharp 2026 reality check: across real systems, the bottleneck is retrieval, not generation. Today's LLMs reason beautifully over good context — so most failures (and most of your effort) are about getting the right chunks into the window, i.e. the indexing + retrieval half of the diagram. That's why this container spends far more time there than on the LLM call itself.

Where It Goes Wrong (Preview)

Because RAG is a pipeline, failures trace to a specific box — which is great news for debugging. A preview (full treatment in Common Failure Modes):

Bad text in → Load & Clean. Mangled PDFs, missing sections.
The right chunk doesn't exist → Chunk. The fact got split across two chunks, so no single chunk answers the question.
The right chunk exists but isn't retrieved → Retrieve / Embed. Wrong k, bad filter, weak embedding model, no reranker.
Too much / irrelevant context → Augment. The signal is buried; the model gets distracted.
Context is right but the answer is wrong → Generate. The model ignored the context or hallucinated anyway (weak grounding instructions).

The debugging discipline: localize the failure to a box before you fix anything. Most teams waste days tuning the LLM prompt when the real bug was a chunk that never made it into the index.

🧪 Try It Yourself

Trace and diagnose — no code needed, just the diagram:

Trace a query. A user asks your docs-bot "what's the refund window?" Name, in order, every box the request passes through from the question to the answer — and say which box turns the question into something searchable.
Localize three bugs. For each symptom, name the single most likely box: (a) the answer cites a competitor's product that isn't in your docs at all; (b) the policy is in your PDFs but the bot says "I don't know"; (c) the right text is clearly retrieved, but the answer contradicts it.

→ (1) Query → Embed (turns the question into a vector — that's the make-it-searchable box) → Retrieve → Augment → Generate → Answer. (2a) Generate — it ignored the context and hallucinated (or there was no relevant context and it didn't refuse). (2b) Retrieve (or Chunk/Embed) — the fact exists but wasn't surfaced: wrong k, bad filter, or it was split across chunks. (2c) Generate — weak grounding; the model overrode the provided context. Notice how naming the box instantly narrows the fix.

Interactive: a RAG pipeline debugger. The full two-phase architecture is drawn as clickable boxes — INDEXING (Load & Clean, Chunk, Embed, Store) above the vector DB and QUERYING (Embed query, Retrieve, Augment, Generate) below it. A symptom appears (garbled PDF text, a fact split across chunks, 'the policy is in the store but the bot says I don't know', 20 chunks burying the signal, an answer citing a competitor not in the docs, wrong-domain retrieval) and the user clicks the single box most likely at fault; the correct box glows green with an explanation, a wrong pick is corrected, and a running score tracks accuracy across all six symptoms. It embodies the lesson's payoff — because RAG is a pipeline, failures trace to a specific box (bad text → Load & Clean, split fact → Chunk, not fetched → Embed/Retrieve, buried signal → Augment, ignores context → Generate) — turning debugging from a mystery into a map. Deterministic.

Mental-Model Corrections

"RAG means fine-tuning the model on my documents." No — nothing about the model changes. RAG retrieves facts at query-time and puts them in the prompt. (That's exactly why it updates instantly when your docs change.)
"It's one pipeline." It's two: indexing (offline) and querying (online), meeting at the vector DB. Confusing them is the #1 beginner error.
"The LLM answers from what it knows." In RAG the LLM should answer only from the provided context — and say "I don't know" when it's absent. That's the whole point of grounding.
"The magic is the LLM / generation." In 2026 the bottleneck is retrieval. Good context → good answer; the hard part is getting good context (chunking + retrieval).
"It's simple — it's just embed and search." Simple to draw; each box is a deep design decision. That depth is the rest of this container.

Key Takeaways

A RAG system is two phases meeting at a vector DB: ① INDEXING (offline) — load → chunk → embed → store; ② QUERYING (online) — embed → retrieve → augment → generate.
The name is the query pipeline: R·A·G = Retrieve · Augment · Generate. Use the same embedding model on both sides.
The LLM answers only from the retrieved context — grounded, cited, and willing to say "I don't know."
RAG is simple to draw, deep to do well. Each box is a design decision; chunking and retrieval are the biggest levers — and in 2026 retrieval, not generation, is the bottleneck.
Failures trace to a box, which makes RAG debuggable. Next, we start building this pipeline for real — beginning with Ingestion (the Load & Clean box).