Fine-Tuning + RAG Together

Introduction

We've spent three lessons treating fine-tuning and RAG as a choice — L184 (Prompting vs RAG vs Fine-Tuning: The Decision) even gave you the rule: knowledge → RAG, behavior → fine-tune. That framing is right for picking your first move. But the most capable systems in 2026 don't pick — they combine.

Here's the reframe: RAG and fine-tuning aren't rivals, they're layers. They solve different problems, so the best stacks use both — RAG for the knowledge layer (fresh, proprietary, auditable facts) and fine-tuning for the behavior layer (how the model uses those facts: faithfully, in your format, with citations). This lesson is how the two fit together — including the named technique that fuses them, RAFT — and where in a RAG system you actually fine-tune.

An infographic titled 'Fine-Tuning + RAG Together', the fourth lesson of the fine-tuning container, showing that retrieval-augmented generation and fine-tuning are not rivals but complementary layers. RAG supplies fresh, proprietary, auditable knowledge at inference; fine-tuning supplies the behavior to use that knowledge well. The picture is a pipeline, query then retriever then context then generator then answer, with two places to fine-tune. RAG alone breaks in two ways: a stock retriever fetches the wrong documents roughly forty percent of the time, and a base generator has no guarantee it adheres to the retrieved context, so it ignores it, will not cite, and drops your format, producing a confident, well-structured, wrong, uncited answer. The first fix point is the retriever, where you fine-tune the embedding model and a reranker for domain recall and precision so the right passages surface. The second fix point is the generator, fine-tuned with RAFT, Retrieval-Augmented Fine-Tuning, where each training example is one golden document plus four distractor documents and the model learns to write chain-of-thought reasoning with verbatim citations from the correct source, so it explicitly identifies which document it is using, ignores the distractors, cites, applies your domain format and refusals, and says when the answer is not in the provided context. RAFT captures the style and behavioral benefits of fine-tuning while keeping the freshness and auditability of retrieval. The order is to start with RAG because it is cheap and fixes the knowledge gap, then fine-tune the generator with RAFT if it mis-uses context, and fine-tune the retriever if recall is the bottleneck, never fine-tuning to memorize the facts themselves. The most capable 2026 enterprise stacks combine all three, RAG for facts, fine-tuning for behavior, and the prompt to orchestrate, each handling the layer it was designed for.

Two Different Jobs → Two Layers

Recall the split from the last three lessons: RAG handles knowledge; fine-tuning handles behavior. That's not a reason to choose between them — it's a reason to stack them, because a real product needs both fresh facts and the right behavior.

Layer	RAG (knowledge)	Fine-tuning (behavior)
Gives you	fresh, proprietary, auditable facts at inference	format, tone, reasoning, how to use the facts
Updates by	re-indexing (instant, free)	re-training (slow, costly)
Fails as	wrong docs retrieved · context ignored	stale facts · hallucinated knowledge

Each one's weakness is the other's strength. Fine-tune without RAG and you get the right form on stale / missing facts (and the temptation to bake facts into weights — the L186 (Bad Reasons to Fine-Tune) trap). RAG without fine-tuning and you get fresh facts the model won't use well. Together: right facts, right form.

Why RAG Alone Breaks (And Fine-Tuning Fixes It)

If RAG supplies the facts, why isn't it enough on its own? Because a RAG pipeline has two joints, and each one breaks in a way only fine-tuning fixes:

1) The retriever fetches the wrong documents. Naive RAG retrieval fails ~40% of the time on hard / niche domains — and when it does, the model produces a confident, well-structured answer grounded in the wrong source. Off-the-shelf embeddings don't know your domain's vocabulary, so the right passage never makes the top-k.

2) The generator ignores the context it was given. This one surprises people: even with perfect retrieval, a base model has no guarantee it adheres to the retrieved context. It leans on its training priors, skips half the passage, won't cite, drops your format, and answers anyway. Excellent retrieval still degrades without a trained generator.

Both failures are behavior problems — exactly fine-tuning's job. So in a RAG system there are two places to fine-tune, one per joint.

The Two Places to Fine-Tune in a RAG System

① Fine-tune the RETRIEVER (for recall & precision). Train the embedding model (and/or a reranker) on your domain so the right passages actually surface. The retriever optimizes recall (find everything relevant); a fine-tuned reranker then reorders the top-k for precision (the most pertinent first, so the best chunks fit the context window). This is what fixes the ~40% wrong-doc problem on niche domains.

② Fine-tune the GENERATOR (for faithful use & behavior). Train the model that reads the retrieved context to use it faithfully — ground its answer in the passage, cite the source, ignore distractor chunks, apply your format / tone / refusals, and say "not in the provided context" when nothing supports an answer. The standard recipe for this is RAFT.

You don't always need both. Start with the cheapest fix for the joint that's actually broken — and the interactive below lets you feel which one each scenario needs.

RAFT — Retrieval-Augmented Fine-Tuning

RAFT (Retrieval-Augmented Fine-Tuning, UC Berkeley) is the technique that fuses the two worlds: it fine-tunes the generator for the open-book domain setting — "here are some retrieved documents, answer using them." It keeps RAG's freshness and auditability while adding fine-tuning's style and behavior. The recipe is clever and simple: each training example is 1 golden document + several distractors, and the target answer is chain-of-thought reasoning with a verbatim citation from the golden source — which forces the model to explicitly pick which document it's using, and why.

# RAFT data: teach the GENERATOR to use retrieved context faithfully.
# Each example = 1 GOLDEN doc + DISTRACTORS; the target answer reasons, then CITES the golden one.
example = {
    "question": "What is our refund window for the Pro plan?",
    "context": [
        {"id": "D1", "text": "Refund Policy v3: Pro & Enterprise - 30 days, no questions asked."},  # GOLDEN
        {"id": "D2", "text": "Shipping & Returns: physical goods may be returned within 90 days."},   # distractor
        {"id": "D3", "text": "Pricing: Pro is $20/mo, billed monthly."},                              # distractor
        {"id": "D4", "text": "Privacy Policy: account data is retained for 24 months."},              # distractor
    ],
    # TARGET = reason about which doc is relevant, then answer with a VERBATIM citation.
    "answer": (
        "Reasoning: this is a refund-policy question. D2 is shipping, D3 pricing, D4 privacy - all distractors. "
        "D1 states the Pro refund policy. "
        "Answer: 30 days. [cite D1: \"Pro & Enterprise - 30 days\"]"
    ),
}

# Train a few hundred-thousand of these as a LoRA (Unsloth/TRL). The generator learns to:
#   - find the GOLDEN doc and IGNORE distractors,   - CITE verbatim,   - apply your format.
# Include ~20% examples where NO doc has the answer -> teaches an honest
#   "that isn't in the provided context" instead of a confident hallucination.
# (LoRA keeps general ability intact - see L186 on catastrophic forgetting; LoRA details in L193.)

The Order: RAG First, Then Fine-Tune What's Broken

Don't reach for the hybrid on day one. Climb in order and only fix the joint that's actually broken:

Start with RAG (off-the-shelf retriever + base generator). It's cheap and fixes the knowledge gap. Measure it.
Generator mis-using context? (ignores passages, won't cite, wrong format, distracted by irrelevant chunks) → RAFT-fine-tune the generator. This is the most common second step.
Retrieval missing the right docs? (the answer was in your corpus but never retrieved) → fine-tune the retriever / reranker for domain recall.
Both broken? Fine-tune both (the holistic approach) — more powerful, more complexity and maintenance.

The one rule that never changes (from L186 (Bad Reasons to Fine-Tune)): never fine-tune to memorize the facts. You fine-tune to use retrieved facts well. The facts stay in the index, fresh and auditable, where you can update them instantly.

See It: The Hybrid Stack Builder

Build the stack yourself. Pick a scenario, then toggle the retriever and generator on the pipeline and watch the capability meters — and the two failure modes — respond. Try the legal assistant with a stock retriever + base generator (watch both failures fire), then turn each fix-point on and see what it buys.

Interactive: the Hybrid Stack Builder. The user builds a RAG system on an arrow-connected pipeline, query then RETRIEVER then context then GENERATOR then answer, and watches what each layer adds. They pick a scenario (support docs, legal assistant on case law, clinical Q&A, general FAQ), then toggle the two fine-tune fix-points: the retriever (off / stock embeddings / fine-tuned retriever and reranker) and the generator (base / RAFT-fine-tuned). Four capability meters update live, fresh facts, right docs found, faithful use without distraction, and domain behavior with citations, and the two RAG failure modes light up when present, the stock retriever grounding about forty percent of answers in the wrong documents on a niche domain, and the base generator ignoring or mis-using even good retrieved context. A maintenance meter rises with each tuned layer to re-train, and a verdict explains the trade-off, from prompt-only goes stale, to fresh facts but the generator will not use them faithfully, to a faithful generator but retrieval misses the docs, to the full hybrid where RAG supplies the knowledge and a RAFT generator uses it faithfully and cites.

The shape the Lab makes obvious: RAG fills the knowledge meters, fine-tuning fills the behavior meters — and neither one alone fills all four.

Why This Matters

"RAG vs fine-tuning" is the wrong question for anything past a prototype. The teams shipping the most capable enterprise AI in 2026 stopped choosing — they layer RAG for facts and fine-tuning for behavior, each handling the job it was built for. Knowing where the two joints break (retrieval and adherence) and which fix-point to tune means you build a system that's fresh, faithful, and cited — instead of one that confidently quotes the wrong document in a beautiful format. That's the difference between a demo and something a regulated business will actually deploy.

🧪 Try It Yourself

Diagnose a hybrid. Take a RAG system you've used or want to build (or use: "a clinical assistant that answers from current treatment guidelines"). Answer: (1) Which knowledge does RAG own (and how often does it change)? (2) Is the retriever likely to miss niche docs — do you need to fine-tune embeddings? (3) Will a base generator use & cite the context, or do you need RAFT? (4) Sketch one RAFT training example for it (a question, one golden doc, two distractors, and a cited answer). Then open the Hybrid Stack Builder, set that scenario, and confirm which fix-points light up.

Mental-Model Corrections

"It's RAG or fine-tuning." For real products it's usually both — different layers. The decision framing (L184) is for your first move, not your final architecture.
"Good retrieval is enough." No — a base generator has no guarantee it adheres to the context it retrieved. It ignores it ~half the time. RAFT trains the use.
"Fine-tuning the generator means teaching it the facts." The opposite — RAFT teaches it to use retrieved facts (cite, ignore distractors), not to store them. Facts stay in the fresh, auditable index.
"There's one place to fine-tune." There are two: the retriever (recall) and the generator (faithful use). They fix different failures.
"RAFT is exotic." It's a data recipe — golden doc + distractors + cited answer — that you train as a normal LoRA. You already have the tools.

Key Takeaways

RAG and fine-tuning are layers, not rivals: RAG = fresh, auditable knowledge; fine-tuning = the behavior to use it (faithful, cited, in your format). The best 2026 stacks combine both.
RAG alone breaks at two joints: the retriever fetches wrong docs (~40% on niche domains), and the base generator has no guarantee it adheres to the context it got.
So there are two places to fine-tune: the retriever / reranker (domain recall & precision) and the generator (faithful use & behavior).
RAFT fine-tunes the generator for the open-book setting: train on 1 golden + distractor docs → chain-of-thought with verbatim citations, so it picks the right doc, ignores the rest, and cites. Include ~20% no-answer examples for honest refusals.
The order: RAG first → RAFT the generator if it mis-uses context → fine-tune the retriever if recall is the bottleneck. Never fine-tune to memorize facts — keep them in the index.

Next: L188 — Estimating Cost & Effort (what a fine-tune — or a hybrid — actually costs in data, compute, and maintenance), closing Section 1 before we open the mechanics in Section 2.