Common RAG Failure Modes

Introduction

You've built the whole pipeline — ingest, chunk, embed, store, retrieve, augment, generate, cite. Now the lesson that separates people who demo RAG from people who run RAG in production: when it gives a wrong answer, how do you find out why?

Here's the liberating truth this whole container has been building toward: RAG fails at the seams, and every failure traces to exactly one box. A wrong answer is never just "the AI messed up" — it's parsing lost the text, or chunking split it, or retrieval missed it, or augmentation buried it, or generation ignored it. Name the box, and the fix becomes obvious. The skill is diagnosis, and most teams do it backwards.

In this lesson you'll learn:

The canonical failure modes (Barnett et al.'s Seven Failure Points) — mapped to boxes
The #1 debugging mistake — fixing the wrong box
A repeatable diagnostic method: follow the answer through the pipeline
Why RAG robustness is evolved through operation, not designed in upfront

Every Failure Traces to a Box

In 2024, Barnett et al. studied real RAG systems and catalogued Seven Failure Points (arXiv:2401.05856). The powerful thing isn't the list — it's that each one lives in a specific box of the pipeline you built. Here's the map:

#	Failure	Box	What happened
1	Missing Content	Retrieve / Ingest	The answer isn't in the corpus — so the system should abstain, but often makes one up.
2	Missed the Top-Ranked	Retrieve	The answer exists but wasn't in the top-k.
3	Not in Context	Augment	Retrieved, but trimmed out of the final prompt (budget/consolidation).
4	Not Extracted	Augment / Generate	In the context, but the model didn't pull it out (noise, lost-in-the-middle).
5	Wrong Format	Generate	Ignored the requested format (list, JSON, …).
6	Incorrect Specificity	Generate / Chunk	Too vague or too detailed for the question.
7	Incomplete	Chunk / Retrieve	Partial answer — info was spread across chunks, not all surfaced.

Add a few the field has learned since: garbled text (Parse), hallucination despite good context (Generate — fix with grounding + verification from last lesson), stale answers (Store/lifecycle — re-index), conflicting sources (rank by recency/authority), and prompt injection via a retrieved chunk (treat retrieved content as untrusted).

An infographic titled 'RAG Failure Modes — Localize Before You Fix'. It says every failure traces to one box, and maps Barnett et al.'s seven failure points onto the pipeline as five cards, each with a warning (the failure) and a checkmark (the fix). PARSE: garbled text, broken tables, empty scans -> use the right parser or OCR and parse to Markdown. CHUNK: failure point 7 Incomplete, the answer is split across chunks -> chunk strategy plus overlap, or parent-document retrieval. RETRIEVE: failure point 1 Missing (not in corpus) and failure point 2 Missed top-k -> add documents or abstain, use hybrid search plus reranking, raise k. AUGMENT: failure point 3 Not in context and failure point 4 Not extracted (lost in the middle) -> rerank, budget the context, reorder with best chunks at the ends. GENERATE: hallucinates, failure point 5 wrong format, failure point 6 wrong specificity -> ground and verify, use structured output. A note adds: stale answers need re-indexing, conflicting sources should be ranked by recency or authority, and injected text in a chunk means treating retrieved content as untrusted. A diagnostic strip says follow the answer through the pipeline with five checks: 1 is the answer even in the corpus, 2 in a single chunk that can answer, 3 in the retrieved top-k (log this first), 4 survived into the final prompt, 5 did the model use it (grounded). A banner reads: most teams rewrite the Generate prompt when the bug is in Retrieve or Chunk; localize first — RAG robustness is evolved through operation, not designed in upfront.

The #1 Debugging Mistake: Fixing the Wrong Box

Watch what almost every team does when an answer is wrong: they rewrite the LLM prompt. They tweak wording, add "be accurate," raise or lower temperature — poking the Generate box for hours.

But look at the map again: most RAG failures are upstream of generation. If the right chunk was never retrieved (#1, #2) or never made it into the prompt (#3), no prompt wording can save you — the model literally never saw the answer. You're polishing the last box while the bug sits three boxes back.

The discipline: localize before you fix. Find the box that's actually broken before you change anything. A failure in Generate and a failure in Retrieve can produce the identical wrong answer on the surface — but the fixes are opposite ends of the pipeline.

This single reframe — which box? before what change? — will save you more time than any other technique in this course.

The Diagnostic: Follow the Answer Through the Pipeline

Localizing is mechanical once you have a method. For a failing query, trace the answer-bearing content forward and find the first box where it falls out:

Is the answer even in the corpus? Grep your parsed text. If it's not there → #1 Missing Content (Ingest). Fix coverage — and make the system abstain instead of guessing.
Is it in a single chunk that can answer? If it's split across two chunks → #7 Incomplete (Chunk). Fix chunking/overlap.
Is the right chunk in the retrieved top-k? → #2 Missed (Retrieve). This is the highest-value check — log what was retrieved.
Did it survive into the final prompt? If reranking/budget dropped it → #3 Not in Context (Augment).
Did the model use it? Right context, wrong answer → #4 Not Extracted or hallucination (Generate). Fix grounding/reordering.

The whole method collapses to one habit: log the retrieved chunks for every failing query. That one log line answers steps 3–5 at a glance and is where ~80% of RAG debugging happens. Here's rag() instrumented to do exactly that:

def rag_debug(question, k=5, where=None):
    """Same rag() — but log the answer's journey through every box."""
    hits = retrieve(question, k=k, where=where)

    print("─" * 60)
    print("Q:", question)
    # ③ THE single most valuable line in RAG debugging: see what was retrieved.
    for i, h in enumerate(hits):
        print(f"  [{i+1}] sim={h.get('score'):.3f}  {h['source']} p.{h['page']}")
        print(f"      {h['text'][:120]}…")

    prompt = build_prompt(question, hits)
    print("④ context chars:", len(prompt))        # did the right chunk survive into the prompt?

    answer = generate(prompt)
    print("⑤ answer:", answer)                     # did the model actually USE the context?
    return answer

Run your failing question through this and the broken box announces itself: empty/irrelevant retrieved list → Retrieve/Chunk/Ingest; right chunks retrieved but wrong answer → Augment/Generate. You stop guessing.

Fix the Right Box (Quick Reference)

Once localized, the fix follows directly from the box — and most are things you've already learned:

Parse — garbled text → better parser / OCR, parse to Markdown (L86).
Chunk — split or incomplete answers → tune size/overlap, semantic or parent-document chunking (next section).
Retrieve — missed the right chunk → hybrid search + reranking, raise k, better embedding model (next section); if the content is genuinely absent, add it and abstain in the meantime.
Augment — dropped or buried context → rerank, budget the window, reorder (best chunks at the ends — lost-in-the-middle) (L87).
Generate — hallucination, format, specificity → grounding + verification and structured output (L88); set temperature to 0.
Lifecycle / security — stale → re-index; conflicts → rank by recency/authority; injected instructions in a chunk → treat retrieved text as untrusted data, never as commands.

Notice the payoff of the whole container: every fix is a tool you already have. Diagnosis just tells you which one to reach for.

A Hard Truth: Robustness Is Evolved, Not Designed

Barnett et al.'s two deepest findings are worth internalizing, because they reshape how you approach a RAG project:

You can only validate a RAG system during operation. You cannot unit-test your way to a robust RAG system before launch — the failure modes only reveal themselves against real, messy, adversarial user queries you didn't anticipate. The corpus you tested on is not the corpus users will probe.
Robustness evolves — it isn't designed in at the start. The best RAG systems aren't the ones architected perfectly on day one; they're the ones with a tight feedback loop: log real queries → spot failures → localize the box → fix → repeat.

This is why the next stages of this course matter so much: evaluation (turn "it feels wrong" into a number) and observability (capture every query's retrieved chunks and final prompt in production). You don't prevent all failures up front — you build the machinery to find and fix them fast. A RAG system is less a thing you build and more a thing you cultivate.

🧪 Try It Yourself

Diagnose, don't guess. For each symptom, name the box (and failure #), the next log you'd check, and the fix:

A user asks about a policy that exists in handbook.pdf, but the bot answers "I don't know."
The bot confidently states a refund window that contradicts your docs; the retrieved chunks (you logged them) clearly contain the correct number.
The answer is correct but half the steps are missing from a multi-step procedure.
Asked for the price, the bot writes three correct paragraphs about pricing but never gives the number.
Your team has spent two days editing the LLM prompt and nothing improves. What should they do instead?

→ (1) Retrieve, #2 Missed top-k — log the retrieved chunks; the right one isn't there. Fix: hybrid + rerank, ↑k, or fix chunking. (2) Generate — the context was right but the model ignored it (hallucination); tighten grounding, set temp 0, add verification (L88). (3) Chunk/Retrieve, #7 Incomplete — steps were split across chunks and not all surfaced; fix overlap / parent-doc / ↑k. (4) Generate, #6 Incorrect specificity — prompt it to answer directly and concisely; check chunk granularity. (5) Stop poking Generate — run the diagnostic trace and localize the box; the bug is almost certainly upstream in Retrieve or Chunk.

Interactive: a RAG diagnostic-trace stepper. The user picks a failing symptom (garbled text, missing steps, an 'I don't know' about a policy that exists, or an answer that contradicts the docs), optionally guesses which pipeline box is at fault, then steps the diagnostic: each check reveals a realistic log line — is the answer in the parsed corpus? in a single chunk? in the retrieved top-k (log the chunks!)? in the final prompt? did the model use it? — and the trace stops at the FIRST check that fails, localizing the bug to one box (Parse, Chunk, Retrieve, or Generate) with its failure number and the concrete fix. If the user's guess was wrong, it calls out the #1 debugging mistake: tuning that box would have wasted hours while the real bug sat upstream. The lesson's method made tangible: follow the answer forward, the first 'no' is your box, and log the retrieved chunks because that check catches ~80% of bugs — localize before you fix. Deterministic.

Mental-Model Corrections

"The AI just gave a wrong answer." There's no single "the AI" — there are boxes, and the failure lives in one of them. Name it.
"Wrong answer → fix the prompt." Most failures are upstream of generation. If the chunk wasn't retrieved or wasn't in the prompt, no prompt edit helps. Localize first.
"I'll log the answer to debug." The answer is the least useful thing to log. Log the retrieved chunks — that's where ~80% of bugs are visible.
"Missing content and bad retrieval look different." On the surface they're identical wrong answers — only tracing the pipeline tells them apart, and they have opposite fixes.
"We'll design it robust up front." You can't — robustness is evolved through operation. Build the feedback loop (eval + observability), not a perfect day-one design.
"Retrieved text is just data." It can carry injected instructions — treat retrieved content as untrusted, never as commands.

Key Takeaways

Every RAG failure traces to one box. Barnett's 7 failure points map cleanly onto Parse → Chunk → Retrieve → Augment → Generate (plus lifecycle & security).
The #1 mistake is fixing the wrong box — endlessly tuning Generate when the bug is upstream in Retrieve or Chunk. Localize before you fix.
Use the diagnostic trace: in corpus? → in a chunk? → in the top-k? → in the prompt? → used? The one habit that matters most: log the retrieved chunks for every failing query.
Each fix is a tool you already have — diagnosis just says which one (parser, chunking, hybrid+rerank, reorder, grounding+verify, re-index).
RAG robustness is evolved, not designed — validated in operation via a tight log → localize → fix loop. That's why evaluation and observability come next.
This closes "Building Your First RAG Pipeline." You can now build and debug an end-to-end RAG system. Next we go deeper — chunking strategies, the single biggest lever on quality.