Case Study: Debugging a Failing AI Feature

Introduction

This lesson puts the whole section to work. Across L171–L175 you learned the moves — look at your data, build a taxonomy, write targeted evals, run the improve loop, gate regressions. Now we debug one real failing feature end-to-end, in order, and add the one skill a list of moves can't teach: localizing the failure — figuring out which stage of a multi-step system actually broke.

That skill is the difference between fixing a bug and guessing at it. When a RAG or agent feature gives a wrong answer, “it's wrong” is an end-to-end verdict that hides the root cause: the same symptom can come from bad retrieval, a model that ignores good evidence, or a prompt that never told it what it needed. Reword the prompt when retrieval was the problem and you ship a change that doesn't move the failure at all. So we'll walk the arc — symptom → traces → taxonomy → localize → fix → re-measure → gate — on a support assistant that “feels off,” and you'll see the same vague complaint resolve into three different stages, each with a different fix.

An infographic titled 'Case Study: Debugging a Failing AI Feature' that walks the entire error-analysis and eval flywheel end to end on one failing feature, a customer-support assistant. The arc runs left to right with arrows: the symptom is that the feature feels off and users say it is wrong even though the dashboard reads eighty-eight percent helpful, because an aggregate end-to-end score hides the root cause; then you look at the data by reading real traces and open-coding the first error in each; then you build a taxonomy and count to get a Pareto of failure modes and pick the most frequent one; then comes the key debugging skill, localize, which pinpoints which stage of the pipeline actually broke; then you match the fix to that stage; then you re-measure the targeted eval; then you add the case to the regression gate so it can never silently return. The centerpiece is the isolation test for localizing a failure: you inject known-good context into the model, and if the answer becomes correct then retrieval was the bug because the model is fine but the right evidence was not fetched, while if the answer is still wrong then generation is the bug because the model is ignoring or contradicting the evidence in front of it, and if injecting good context changes nothing then it is a prompt or specification gap such as the system prompt never giving the model today's date. The same vague symptom, it is wrong, resolves to three completely different stages across three cases: a refund question where retrieval fetched a semantically similar distractor document instead of the refund policy, fixed by better chunking, hybrid search, or reranking, taking the eval from forty-one to ninety-three percent; a warranty question where the correct document was retrieved but generation hallucinated a yes against an explicit exclusion, fixed by a faithfulness guardrail that verifies grounding, taking the eval from fifty-five to ninety percent; and a scheduling question where tomorrow resolved to the wrong day because the prompt lacked the current date, fixed by injecting today's date or a date tool, taking the eval from thirty-four to ninety-five percent. The guiding rule is to stop at the most upstream error in the trace because it is usually the causal root of the downstream problems, and to localize rather than guess because fixing the wrong stage ships a change that does not move the failure. The takeaway banner reads: read the trace, localize the broken stage with the isolation test, match the fix to that stage, re-measure, and pin the case as a regression test.

The Symptom: “It Feels Off”

Our feature: a customer-support assistant (RAG over the company's help docs, plus a couple of tools). The signal that something's wrong is the one every team starts with — vibes. Support tickets are creeping up; a PM forwards a screenshot of a confidently wrong answer; the on-call engineer “feels like it got worse.”

Meanwhile, the dashboard says 88% helpful. This is the trap from L171 (Look at Your Data — The #1 Rule): an aggregate, average-y metric hides the failures that matter — most answers really are fine, so the average stays green while a specific, damaging mode runs rampant. “It feels off” and “the dashboard is green” are both true at once, and neither tells you what to do.

The instinct here is to start guessing fixes — “let's tweak the prompt,” “maybe switch models,” “add more docs.” Resist it. You don't yet know what's broken. The first move is never a fix; it's to look.

Step 1 — Instrument, Then Read the Traces

You can't debug what you can't see, so first instrument the feature to log full traces — not just input→output, but the whole multi-turn record: the user's message, the retrieved documents (with scores), the assembled prompt, every tool call, and the final answer (this is the tracing you'll formalize in L177 (Tracing LLM & Agent Calls)).

Then do the highest-ROI thing in all of AI engineering: read ~50–100 real traces and open-code them — one plain-language sentence per error (L171, L172 (Open Coding & Failure Taxonomies)). The one rule that makes this fast and correct:

Stop at the most upstream error in the trace. In a pipeline, failures cascade — the first thing that went wrong is usually the causal root, and fixing it makes the downstream noise disappear.

Real traces are messier than your test cases — Hamel Husain's canonical example is a user typing “Hello there, what's up to four month rent?” The note isn't “model is dumb”; it's specific: “ambiguous intent — should have asked a clarifying question.” That specificity is what makes the next step work.

Step 2 — Taxonomy & Counting

A pile of notes isn't a plan. Cluster the open codes into a handful of named, binary failure modes and then do the deceptively powerful thing: count them (L172). As Hamel Husain puts it, “counting is powerful” — frequencies turn a wall of anecdotes into a Pareto chart that tells you, with confidence, what to work on first.

For our support assistant, the counts come back like this:

Failure mode	Share of errors
Wrong / invented answer (the content is just incorrect)	~45%
Hallucinates despite having the right doc	~20%
Mis-resolves dates / times (“tomorrow”)	~15%
Wrong format / too long	~12%
Other	~8%

The #1 mode — “wrong / invented answer” — is where you start (descend the Pareto, not the scariest-sounding one). But here's the catch that makes this a debugging lesson and not just a recap: “wrong / invented answer” is a symptom, not a cause. Three different broken stages can all produce it. Before you can fix it, you have to localize it.

Step 3 — Localize the Failure (the Isolation Test)

This is the heart of debugging a multi-step feature. End-to-end, all you know is “the answer was wrong.” You need to find which stage of query → retrieve → assemble prompt → generate → answer actually broke — because each stage has a completely different fix. The technique is a clean isolation test:

Hand the model known-good context (the correct document, manually) and re-run generation.

The answer becomes correct → the model was fine all along; retrieval was the bug (it didn't fetch the right evidence).

The answer is still wrong → retrieval was fine; generation is the bug (it's ignoring or contradicting the evidence).

Injecting context changes nothing → it's not a knowledge problem at all; the prompt/assembly is the gap (a buried rule, the wrong context, no current date).

This single move splits the failure space and tells you exactly where to look. Retrieval and generation fail differently — retrieval by missing the evidence or ranking a distractor too high, generation by ignoring evidence or inventing claims — and you must not confuse them. Here it is in code:

# The core debugging move for a multi-step feature: LOCALIZE the failure.
# "It's wrong" end-to-end hides WHICH stage broke. Isolate generation from retrieval.

def localize(app, case):
    # 1) Is the right evidence even retrieved?
    docs = app.retrieve(case.query)
    if not contains(docs, case.gold_evidence):
        return "RETRIEVAL"     # the answer isn't in what we fetched -> fix retrieval

    # 2) Retrieval is fine -> does generation USE the evidence when handed it cleanly?
    answer = app.generate(case.query, context=case.gold_evidence)   # inject known-good context
    if not is_correct(answer, case.gold_answer):
        return "GENERATION"    # ignores / contradicts the evidence -> fix grounding

    # 3) Good context -> good answer, yet it still fails live? the prompt/assembly is the gap
    return "PROMPT"            # e.g. no current date, a buried rule, wrong context assembled
# The test in one line: provide known-good context. Answer FIXES -> retrieval was the bug.
# Still WRONG -> generation. Changes NOTHING -> the prompt. Don't guess; localize.

More generally, observe every span of the trace — the retrieved docs and their scores, the assembled context, the generated answer — because end-to-end accuracy hides root causes. When you have a failing trace, compare it against a successful one and find where the behavior diverges. That divergence point is your bug.

See It: The Attribution Lab

Now debug it yourself. Below are three failing cases from the support assistant — same symptom (“the answer is wrong”), but each breaks at a different stage. For each: inspect the spans of the trace, run the isolation test (inject known-good context), attribute the failure to the right stage, then pick the matching fix. Guess the stage wrong and you'll see the fix miss — localize, don't guess.

Interactive: a failure-attribution lab on a customer-support RAG assistant whose dashboard reads 88% 'helpful' but whose answers are wrong. Three failing cases as tabs, each the SAME symptom ('the answer is wrong') but breaking at a DIFFERENT stage. For each case the user: (1) inspects the trace as a clickable pipeline with arrow connectors — QUERY ▸ RETRIEVE ▸ ASSEMBLE ▸ GENERATE ▸ ANSWER — clicking each span reveals its real output (e.g. RETRIEVE shows which doc came back with what similarity score); (2) runs the ISOLATION TEST ('inject the correct context & re-run'), which reports whether the answer fixed; (3) attributes the failure to Retrieval / Generation / Prompt-spec (wrong picks get a hint pointing back to the isolation result); (4) on a correct attribution, picks the matching fix from four options (fix retrieval / faithfulness guardrail / add today's date / fine-tune). Case 1 (refund question): retrieval fetched a semantically-similar DISTRACTOR ('Shipping & Returns', sim 0.71) and the real Refund Policy ranked #6 — injecting the correct doc fixes the answer, so it's RETRIEVAL → fix chunking/hybrid/rerank, eval 41%→93%. Case 2 (warranty/water damage): the CORRECT doc was retrieved (sim 0.93) but generation answered 'Yes, covered' against an explicit exclusion — injecting the doc again still fails, so it's GENERATION → add a faithfulness guardrail, eval 55%→90%. Case 3 ('book a demo for tomorrow'): no retrieval needed; the system prompt has no current date so 'tomorrow' resolves wrong — injecting docs changes nothing, so it's a PROMPT/specification gap → add today's date (or a date tool), eval 34%→95%. Resolving all three shows a recap: same symptom, three stages — you can't fix what you haven't localized. Teaches the isolation test (good context in: answer fixes → retrieval was the bug; still wrong → generation; no change → prompt), span-level inspection, and matching the fix to the localized stage.

Notice the punchline across the three cases: the identical complaint — “it gave a wrong answer” — came from retrieval (a distractor outranked the right doc), then generation (the model ignored a correct doc), then the prompt (it never knew today's date). Three different stages, three different fixes. An engineer who “just tweaks the prompt” fixes one of three and ships the other two.

Step 4 — Match the Fix to the Stage

Once you've localized the stage, the fix follows directly — the three gulfs decision from L174 (The Analyze → Measure → Improve Loop), now pointed at a specific span:

Retrieval broke (the refund case): the evidence exists but wasn't surfaced → fix retrieval — better chunking, hybrid search (keyword + vector), or a reranker so the right doc beats the distractor. Not the prompt.
Generation broke (the warranty case): the right doc was right there and the model overrode it → fix grounding — instruct “answer only from the source and cite it,” and add a faithfulness guardrail that verifies each claim is supported and blocks it if not (L175 (Catching Regressions Before They Ship) ideas applied as a runtime check).
Prompt/spec broke (the “tomorrow” case): it was never told what it needed → patch the prompt (inject today's date) or give it a date tool. The cheapest rung, and a stunningly common root cause — “even two incorrect words in a monster system prompt can significantly degrade quality.”

And the discipline from the loop holds: change one thing at a time. You localized one stage; fix that stage; then re-measure. Fine-tuning, notably, is on none of these lists — it's the rare last resort, not the reflex.

Step 5 — Re-Measure, Then Gate It

A fix isn't done when it looks better — it's done when the number moves and stays moved. Two steps close the loop:

Re-measure the targeted eval. You built a binary eval for this exact mode (L173 (From Failures to Targeted Evals)); run it before and after. In the lab, the refund (retrieval) fix takes that eval 41% → 93%; the warranty (generation) fix, 55% → 90%; the date (prompt) fix, 34% → 95%. Now you can prove the fix worked — and check that nothing else regressed.
Add the case to the regression set. Pin the exact failing trace as a permanent test before you ship the fix (L175 — the ratchet). That one bug can now never silently return, and your baseline only climbs.

Then the loop turns: the #2 mode is waiting on the Pareto, and production keeps feeding new traces into the next round of analysis (L174). Here's the whole debugging loop in one function:

# Debugging a failing feature = the eval flywheel (L171-175) aimed at one symptom.
def debug_feature(app, suite):
    traces = sample_traces(app, n=100)            # 1. LOOK: real, full multi-turn traces
    notes  = [first_failure(t) for t in traces]   # 2. open-code the FIRST (most upstream) error
    modes  = taxonomy(notes)                       # 3. cluster + COUNT -> a Pareto of modes
    top    = modes[0]                              # 4. the most frequent mode (not the scariest)

    stage  = localize(app, top.example)            # 5. LOCALIZE with the isolation test
    fix    = {"RETRIEVAL":  fix_retrieval,         #    match the fix to the broken STAGE
              "GENERATION": add_grounding_guardrail,
              "PROMPT":     patch_system_prompt}[stage]
    apply(app, fix)                                # 6. ONE change, at the root cause

    before = suite.score(top.eval, baseline_app)   # 7. RE-MEASURE the targeted eval
    after  = suite.score(top.eval, app)
    suite.add_regression_case(top.example)         # 8. RATCHET: pin it so it can't return (L175)
    return after - before                          # the lift from one disciplined pass

That's the entire method on one screen: look → localize → fix the root cause → re-measure → ratchet. No new framework, no heroics — just the flywheel, aimed at one symptom and run with discipline.

🧪 Try It Yourself

Take one real complaint about your own feature through the full arc this week:

Pull the trace of a genuinely-wrong output — with the retrieved docs, scores, and assembled prompt, not just the answer.
Localize it. Run the isolation test: hand the model the correct context by hand and re-run. Did the answer fix (retrieval bug), stay wrong (generation bug), or not change (prompt/spec bug)?
Match the fix to the stage — retrieval (chunking/hybrid/rerank), generation (grounding + a faithfulness check), or prompt (add the missing instruction/date/tool). One change.
Re-measure with a binary eval for that mode (build one if you don't have it — L173), before vs. after.
Pin the case into your regression set so it can't come back.

Do this once and the method stops being abstract. You'll have turned “it feels off” into “retrieval was fetching a distractor on refund questions; we added a reranker; the eval went 41%→93%; it's now a regression test.” That sentence is the whole job.

Mental-Model Corrections

“The answer's wrong, so the model is bad.” “Wrong” is an end-to-end symptom, not a cause. Localize the stage — retrieval, generation, or prompt — before you touch anything.
“Just tweak the prompt.” That only helps if the bug is in the prompt/spec. If retrieval fetched the wrong doc, no prompt edit will save you. Run the isolation test first.
“The dashboard is 88%, we're basically fine.” Aggregate metrics hide rare-but-damaging modes. Read the traces; the green average and the real failure coexist.
“Fix every error in the trace.” Stop at the most upstream error — it's usually the root, and fixing it clears the cascade. Don't chase downstream symptoms.
“It hallucinated, so improve the model / fine-tune.” First check retrieval: a hallucination “despite the right doc” is a generation/grounding bug (guardrail + citation); a hallucination because the wrong doc was fetched is a retrieval bug. Different fixes; fine-tuning is rarely either.
“Looks better, ship it.” Not done until the targeted eval moves and the case is a regression test. Vibes got you into this; a number gets you out.

Key Takeaways

Debugging a failing AI feature = the whole flywheel aimed at one symptom: look → taxonomy → localize → fix → re-measure → gate. The new skill is localization.
Start by looking, not fixing. Instrument full traces, read 50–100, open-code the most upstream error (it's the root cause), and count into a Pareto (L171, L172).
Localize with the isolation test: hand the model known-good context. Answer fixes → retrieval was the bug; still wrong → generation; no change → the prompt/spec. End-to-end accuracy hides the stage — don't guess.
Match the fix to the stage: retrieval → chunking/hybrid/rerank; generation → grounding + a faithfulness guardrail; prompt → add the missing instruction/date/tool (L174). One change. Fine-tuning is the rare last resort.
Close the loop: re-measure the targeted eval (L173) to prove the fix and catch regressions, then pin the case as a permanent regression test (L175). Same vague symptom, three different stages — you can't fix what you haven't localized.
This closes the error-analysis section. Next we make the “instrument the traces” step first-class: production observability (L177 (Tracing LLM & Agent Calls)).