Prompting vs RAG vs Fine-Tuning — The Decision

Introduction

Welcome to the fine-tuning part of the course — and we're going to open it with the most important fine-tuning lesson of all: when not to. Before you train a single model weight, you have to win an argument with yourself, because the instinct “our AI isn't good enough — let's fine-tune it” is, the large majority of the time, the wrong move. A startup in a now-famous story “burned $20k fine-tuning what a simple prompt fixed.”

There are exactly three ways to improve an LLM application — prompting, RAG, and fine-tuning — and they solve different problems. This lesson is the decision framework that tells you which one you actually need. Two ideas decide almost every case: a cheap-first ladder (try the fast, free thing before the slow, expensive one) and a single axis — knowledge vs. behavior — that, once you internalize it, prevents the most common and most costly mistake in applied AI. Get this right and the rest of the container (how to fine-tune well) is time well spent. Get it wrong and you'll spend weeks and thousands of dollars solving the wrong problem.

An infographic titled 'Prompting vs RAG vs Fine-Tuning, The Decision', the opener of the fine-tuning container, framing when to reach for each technique. There are three ways to improve an LLM application. Prompt engineering refines the instructions and few-shot examples in context without changing the model weights, and it is the fast, cheap first resort. RAG, retrieval-augmented generation, injects current or proprietary knowledge at inference by retrieving documents and grounding the answer, which is fresh, citable, and updated by re-indexing rather than re-training. Fine-tuning adjusts the model weights by training on hundreds to thousands of labeled examples and is for changing behavior, the format, tone, task, and decision logic, not for adding knowledge. The core split that decides everything is knowledge versus behavior: a missing-facts problem is a RAG problem, a wrong-behavior problem is a fine-tuning problem, and prompting addresses both first. The single most important and most violated rule is that fine-tuning does not reliably add knowledge: it only pattern-matches facts, suffers catastrophic forgetting where it loses prior abilities, still hallucinates outside its new area, and goes stale because the training data is frozen, so knowledge belongs in retrieval, not in the weights, and teams routinely waste money fine-tuning what a prompt or RAG would have fixed. You climb a cheap-first ladder: start with prompting, which takes hours and is nearly free; escalate to RAG when you need real-time or proprietary facts, at modest infrastructure cost; and only fine-tune when you need deep behavioral specialization at scale, which costs the most in data, money, and weeks of work, and is the last rung, not the first. Only climb as high as you must. The best production systems combine all three: RAG supplies the facts, fine-tuning shapes the behavior, and the prompt orchestrates both per request, and the corrected production traces from your data flywheel are exactly the labeled examples that fine-tune the behavior. The takeaway banner reads: knowledge goes to RAG, behavior goes to fine-tuning, prompt first always; fine-tuning does not add knowledge; climb the cheap-first ladder and combine all three.

The Three Techniques

Each technique changes a different layer of the system:

Prompt engineering — refine the instructions and few-shot examples you put in context. No training, no weights changed. “Better specifying what you're looking for.” Hours, nearly free, and it's the first thing you reach for. Limited by the context window and the base model's capability.
RAG (Retrieval-Augmented Generation) — at inference, retrieve relevant documents and inject them into the prompt so the model answers from real, current, proprietary information instead of its frozen memory. It grounds answers (fewer hallucinations), is citable, and you update it by re-indexing — no retraining. Adds a retrieval hop and some infra.
Fine-tuning — actually train the model's weights on hundreds to thousands of your examples, baking a behavior into the model. “Specializing a model.” It can make outputs more consistent and let you use shorter prompts / a smaller model (lower latency & cost at scale). But it's the most expensive path — real data, real money, weeks of work — and, crucially, it changes how the model behaves, not what it knows.

That last clause is the whole ballgame.

The Core Split: Knowledge vs Behavior

Here is the one distinction that resolves most of the decision. Ask: is my problem a KNOWLEDGE gap or a BEHAVIOR gap?

Knowledge gap — the model doesn't know a fact it needs: today's price, your internal policy, a document that postdates its training cutoff. “RAG handles knowledge — current documents, proprietary data, citations.” → RAG. (Or, for a small static set, just put it in the prompt.)
Behavior gap — the model knows enough but won't act right: it ignores your format, drifts off your brand voice, fumbles a domain-specific task or decision pattern no matter how you prompt it. “Fine-tuning shapes behavior, tone, and decision logic.” → fine-tuning.

And prompting addresses both — first. A clearer instruction often fixes a behavior gap; pasting the relevant doc often fixes a knowledge gap. You only escalate when prompting genuinely can't.

The slogan to memorize: knowledge → RAG, behavior → fine-tune, prompt → both, first. “Behavioural change, not knowledge change, is where fine-tuning delivers reliable ROI.” If you can state your problem as “it doesn't know X,” you have a retrieval problem, not a training one — which leads directly to the single biggest mistake.

Fine-Tuning Does NOT Add Knowledge

This is the $20k mistake, and it's worth stating as bluntly as the research does: fine-tuning does not reliably teach a model new facts. When you train on “the refund window is 30 days,” the model learns the pattern and phrasing, not a durable, retrievable fact — “knowledge gains are often limited to surface pattern matching rather than genuine knowledge integration.” Four failure modes follow, all documented:

It pattern-matches, it doesn't integrate. Ask the same fact a slightly different way and it misses.
Catastrophic forgetting. Training on new data degrades prior abilities and even instruction-following — you can make the model worse at things it used to do (arXiv 2308.08747).
It still hallucinates. “A fine-tuned model can still hallucinate if asked a question outside its new area” — statistical priors override facts.
It goes stale. The fact is frozen into the weights; when the policy changes, you must re-train. RAG, you just re-index.

So: facts belong in context (a prompt) or retrieval (RAG) — never baked into the weights. If your goal is “the model should know our latest docs,” the answer is RAG, full stop. Reach for fine-tuning only when the gap is genuinely behavioral — and even then, RAG often supplies the facts the fine-tuned behavior operates on.

The Cheap-First Ladder

Beyond which technique, there's an order: always climb from the cheapest, fastest option and stop the moment it works. The ladder:

Prompt engineering — “start here.” Hours, ~free. A better system prompt, clearer instructions, a few good examples. For a knowledge set under ~200k tokens, full-context prompting is often cheaper and faster than building a RAG pipeline. Astonishing how often this is the whole fix.
RAG — when you need real-time or proprietary facts that don't fit (or shouldn't live) in a static prompt. Modest infra cost (~$70–1,000/month range), a retrieval hop of latency.
Fine-tuning — when you need deep behavioral specialization at scale that prompting can't reach. The most expensive rung: hundreds–thousands of labeled examples, real training cost, weeks of work, and ongoing serving cost.

The discipline: only climb as high as you must. Each rung up is more power but far more cost and maintenance. Most teams that “need to fine-tune” haven't actually exhausted rung 1. Here's the decision as code:

# The decision in one function. Climb the CHEAP-FIRST ladder; split KNOWLEDGE from BEHAVIOR.
def choose(p):
    if not p.tried_a_real_prompt:
        return "PROMPT"     # most "I need to fine-tune" is a better prompt. Free, hours. Do this FIRST.

    if p.gap == "knowledge":            # the model is missing FACTS
        return "RAG"        # retrieve them at inference -> fresh, citable, update by re-indexing.
        # NOT fine-tuning: it only pattern-matches facts, suffers catastrophic forgetting, goes stale.

    if p.gap == "behavior":             # format / style / task / decision-logic a prompt can't hit
        return "FINE_TUNE" if p.has_labeled_data else "BUILD_DATASET_THEN_FINE_TUNE"

    return "HYBRID"         # need new FACTS *and* new BEHAVIOR -> RAG (facts) + fine-tune (behavior)
# Effort & cost climb each rung: prompt (hours, ~free) -> RAG ($ infra) -> fine-tune ($$$, weeks, data).
# Only climb as high as you must — and never fine-tune to "add" knowledge.

Notice the first branch: if you haven't tried a real prompt, the answer is always prompt — regardless of the rest. You earn the right to climb. The cheap rung is not a consolation prize; it's the correct first answer more often than not.

See It: Route Your Problem

Run your own problem through the decision. Tell the router what's missing (knowledge or behavior), whether that knowledge is dynamic/proprietary, whether you've actually tried a strong prompt, and whether you have labeled data — and watch it route you to prompt / RAG / fine-tune / hybrid, with the reasoning, while the comparison matrix highlights the winning column. Try answering “knowledge” and reaching for fine-tuning — the router will stop you.

Interactive: a prompt-vs-RAG-vs-fine-tune decision router. The user sets four inputs via segmented controls: the gap (knowledge / behavior / both), whether that knowledge is dynamic-or-proprietary vs static-general, whether they've tried a strong prompt + few-shot already (yes-still-fails / not-really), and whether they have 100s-1,000s labeled examples (yes / no). The router live-recommends: not-tried-prompt → 'Start with PROMPTING' (most fine-tuning needs are a better prompt); knowledge + dynamic → 'Use RAG' WITH a red warning 'Do NOT fine-tune for this — it pattern-matches facts, catastrophic forgetting, still hallucinates, goes stale; knowledge → RAG'; knowledge + static → 'Prompt it (full-context) or light RAG'; behavior + has-data → 'Fine-tune'; behavior + no-data → 'Build the dataset first then fine-tune (the flywheel)'; both → 'Hybrid: RAG + fine-tune + prompt'. Below, a 3-way comparison matrix (Prompting / RAG / Fine-tuning) across rows — fixes, changes weights?, knowledge (in-context / retrieved-fresh-citable / baked-in-stale-lossy), data needed (0-few / a corpus / 100s-1000s), cost, latency, to update (instant / re-index / re-train), when (always FIRST / need facts / behavior-@-scale LAST) — with the recommended column(s) glowing. Teaches: knowledge → RAG, behavior → fine-tune, prompt first always, fine-tuning ≠ adding knowledge, and the cheap-first ladder; the matrix makes the tradeoffs concrete.

Flip the inputs and the answer moves — because it should. Say you haven't tried a prompt and everything routes to prompt first. Say the gap is knowledge and the router refuses to fine-tune, every time. Say behavior with no data and it tells you to build the dataset first (your flywheel from L183). The framework isn't a vibe; it's a function of your actual constraints.

Combine Them: The Hybrid

These aren't rival camps — the best production systems in 2026 use all three, each doing the job it's good at: “RAG delivers current facts at runtime, fine-tuning shapes behavior and tone, and prompt engineering orchestrates both and controls output per request.”

A mature stack looks like this: a fine-tuned model that reliably produces your format/tone/task behavior, fed by RAG that supplies the fresh, citable facts, assembled and steered by a prompt per request. Knowledge stays in retrieval (so it's always current); behavior lives in the weights (so it's consistent and cheap to invoke); the prompt glues them together.

# The best 2026 production stacks COMBINE all three — each doing the job it's good at.
def answer(query):
    docs   = retrieve(query)                       # RAG  -> the FACTS (fresh, citable, re-indexable)
    prompt = orchestrate(system, query, docs)      # PROMPT -> assemble context + control output
    return ft_model.generate(prompt)               # FINE-TUNED model -> the BEHAVIOR (tone/format/task)

# RAG supplies knowledge · fine-tuning supplies behavior · the prompt orchestrates per request.
# And the corrected production traces from your data flywheel (L183) ARE the labeled examples that
# fine-tune the behavior. Close the loop, then climb the ladder deliberately — not by reflex.

And the loop closes back to the previous container: the corrected production traces from your data flywheel (L183 (Closing the Loop: Production → Dataset)) are exactly the labeled examples a behavioral fine-tune needs. You don't guess your way to fine-tuning data — you harvest it. Which is why we did evaluation and observability first: you can't fine-tune well without the dataset, the evals to know it worked, and the monitoring to catch when it drifts. The rest of this container is how to fine-tune well — now that you know when.

🧪 Try It Yourself

Before you fine-tune anything, run the gauntlet on a real problem:

State the gap in one sentence. Does the model not know something (“it doesn't know our refund policy”) or not behave right (“it won't output our JSON format reliably”)? That single classification picks RAG vs fine-tune.
Try a real prompt first — actually. Spend an hour on a strong system prompt + 3–5 good few-shot examples. Measure it (your evals from the last container). Most “fine-tuning” needs die here.
If it's knowledge → reach for RAG, never fine-tuning. Put the docs in context or behind retrieval. Confirm: is the failure “wrong facts” (RAG) or “right facts, wrong form” (fine-tune)?
If it's behavior and a prompt truly can't do it → check your data. Do you have 100s–1,000s of labeled examples? If not, you're not ready — keep prompting and harvest corrected traces (L183) until you are.
Compute the cost of the next rung before you climb it: prompt (hours) vs RAG ($ + a hop) vs fine-tune ($$$ + weeks + maintenance). Climb only if the cheaper rung genuinely failed.

Do this honestly and most of the time you'll save the $20k — and the times you do fine-tune, you'll be fine-tuning the right thing.

Mental-Model Corrections

“Our AI is weak — let's fine-tune.” Usually the wrong reflex. Most quality gaps are fixed by a better prompt or RAG. Earn the climb: prompt → RAG → fine-tune.
“Fine-tune so the model knows our docs.” The $20k mistake. Fine-tuning doesn't reliably add facts — it pattern-matches, forgets (catastrophic forgetting), still hallucinates, and goes stale. Knowledge → RAG.
“RAG vs fine-tuning — pick one.” They solve different problems (knowledge vs behavior) and the best stacks combine them: RAG for facts, fine-tune for behavior, prompt to orchestrate.
“Fine-tuning makes the model smarter.” It makes it more consistent at a behavior, often with shorter prompts — but it can make it worse at other things (forgetting). It's specialization, not intelligence.
“We need fine-tuning data — let's write some.” Hand-written data is brittle. Harvest corrected production traces from your flywheel (L183); they're the real distribution.
“Prompting is the beginner option.” Prompting is the correct first answer most of the time — and full-context prompting (< ~200k tokens) often beats a RAG pipeline on cost and speed. Cheap-first is a discipline, not a limitation.

Key Takeaways

Three techniques, three jobs: prompting (specify behavior in-context, free/instant), RAG (inject missing/dynamic knowledge at inference, re-indexable), fine-tuning (bake a behavior into the weights, needs data + cost).
The core split decides most cases: knowledge → RAG, behavior → fine-tune, prompt → both, first. “Behavioural change, not knowledge change, is where fine-tuning delivers ROI.”
Fine-tuning does NOT add knowledge — it pattern-matches, suffers catastrophic forgetting, still hallucinates, and goes stale. Facts belong in context or retrieval, never the weights. This is the most expensive, most common mistake.
Climb the cheap-first ladder: prompt (hours, ~free) → RAG (facts, $ infra) → fine-tune (behavior @ scale, $$$ + weeks). Only climb as high as you must; full-context prompting (< ~200k tokens) often beats RAG.
The best stacks combine all three: RAG (facts) + fine-tuning (behavior) + prompting (orchestration). The corrected traces from your data flywheel (L183) are the labeled examples a fine-tune needs.
Now that you know when, the rest of this container is how: next, the good reasons to fine-tune (L185 (Good Reasons to Fine-Tune)) — the behavioral wins where it genuinely pays off.