Open Coding & Failure Taxonomies

Introduction

You followed the #1 rule (L171 (Look at Your Data (The #1 Rule))) and read your traces. Now you're holding a pile of messy notes — “invented an HOA fee,” “suggested a place over budget,” “used the wrong tool.” Useful, but unstructured: you can't act on a heap. This lesson is the method that turns that heap into a map — a structured, counted, prioritized failure taxonomy that tells you exactly which evals to build and in what order.

The method isn't invented for AI; it's borrowed wholesale from qualitative research — grounded theory's open coding and axial coding — which is why the people who do evals best (Hamel Husain, Shreya Shankar) describe it as “data science ideas applied to AI.” The whole point is to let the categories emerge from the data, bottom-up, instead of imposing a checklist you guessed at a desk. Three moves: label each failure, cluster the labels into modes, count the modes. The count is the magic — it reveals that a handful of failure modes cause most of your problems, and that's your roadmap.

An infographic titled 'Open Coding & Failure Taxonomies' about turning the messy pile of error-analysis notes from reading traces into a structured, counted, prioritized failure taxonomy, using the qualitative research method called grounded theory applied to AI. The first step is open coding: for each trace you write a short descriptive label, a code, for the first thing that went wrong, in your own words and with no predefined categories, letting the failures define themselves bottom-up. The second step is axial coding: you cluster similar codes into higher-level named categories, the failure modes, so for example proposed an unavailable showing time and suggested a property over budget both cluster under a single mode called constraint violation. Aim for roughly five to ten modes, each defined to be binary, observably yes or no, with clear definitions, and as distinct as possible so they do not overlap. An LLM can propose the initial clustering through a careful prompt, but a human reviews, renames, merges, and refines, because the model is a starting point, not the final answer. The third step is to re-label every trace with the finished taxonomy and count the frequency of each mode in a pivot table, which transforms qualitative insight into a quantitative roadmap and almost always reveals a Pareto distribution where just three or so modes account for most of the failures, giving a clear data-driven priority: fix the biggest mode first, not the scariest-sounding one. Coding is iterative, not one-pass: by the constant-comparison principle you re-code from scratch as your own understanding of the criteria shifts, a kind of criteria drift, and you stop at theoretical saturation when new traces stop producing new modes. Pitfalls to avoid are too many categories from over-splitting, overlapping categories that double-count, premature top-down categories that blind you to real failures, and an unrepresentative sample that misses important modes. The payoff is that each binary failure mode is already an evaluation specification, a yes-or-no question you can turn into a targeted eval, and the Pareto count tells you which ones to build first, with a gold-standard set of about a hundred personally pass-fail-labeled examples for the top mode. The takeaway banner reads: open code each trace, axial code into five to ten binary failure modes, count them into a Pareto, and fix the biggest first, because a handful of modes cause most of your failures and each one becomes an eval.

From Mess to Map: Grounded Theory

Grounded theory is an inductive method: instead of starting with a theory and looking for evidence, you start with raw observations and let the structure emerge from them. Applied to AI error analysis, it's three iterative phases:

Open coding — segment your raw notes into short descriptive labels (“codes”).
Axial coding — group related codes into higher-level categories (your failure modes).
Count — re-label every trace with the finished categories and tally the frequencies.

Why bottom-up? Because top-down blinds you. If you walk in with a pre-built list of categories (“hallucination, toxicity, latency…”), you'll dutifully sort failures into those buckets and never notice the weird, product-specific failure mode that's actually killing you — the one no generic taxonomy has a slot for. The discipline of this lesson is to suspend your assumptions and let the failures name themselves. (“Open coding” here means qualitative labeling — nothing to do with software code.)

Open Coding: Label What You See

Open coding is what you already started in L171: read a trace, and write a short, concrete label for the first thing that went wrong — in your own words, one observation per code. Not “bad output” but “proposed a showing time outside business hours.” Not “hallucination” (that's already a category — too early) but “invented an HOA fee that isn't in the listing.”

Three habits make open codes useful:

Stay descriptive, not categorical. Describe the specific failure you see; the categories come later, in axial coding. Jumping to “it hallucinated” on trace 1 pre-commits you to a taxonomy before you've seen the data.
One code per observation, the first failure (the cascade rule from L171).
Constant comparison. As you code trace 30, glance back at traces 1–29 — does this failure match one you've already labeled, or is it genuinely new? This keeps your language consistent and surfaces the recurring patterns.

The output is a column of free-text codes, one per trace — raw, redundant, in your voice. That redundancy is the signal: the labels you write over and over are your failure modes trying to announce themselves.

Axial Coding: Cluster Into Failure Modes

Axial coding is the synthesis: you take the pile of open codes and group similar ones into named, higher-level categories — your failure modes. “Proposed an unavailable showing time,” “suggested a property over budget,” and “recommended a ruled-out neighborhood” are three different codes that share one essence — the model broke a stated constraint — so they collapse into a single mode: Constraint Violation.

Four rules for good failure modes:

Aim for ~5–10. Fewer and they're too coarse to act on; many more and you've over-split (real taxonomies land here — MAST found 14 modes, ErrorAtlas 17 categories).
Make each one binary. Define the mode so it's observably yes/no — “did the output violate a stated constraint? y/n.” Binary modes have crisp definitions and (next lesson) become evals directly.
Minimize overlap, cover the data. Distinct enough that a failure lands in one obvious mode, exhaustive enough that the modes together cover what you saw (the MECE instinct — don't double-count, don't leave failures homeless).
Let Claude assist, but you own it. An LLM is great at proposing an initial clustering of your notes; it is not the final word.

from anthropic import Anthropic
import json
client = Anthropic()

# AXIAL CODING, assisted. You wrote open-ended notes (L171); Claude PROPOSES clusters as a
# starting point. You then rename, merge, split, and OWN the final taxonomy — it's not the answer.
def suggest_taxonomy(open_codes):              # open_codes: list[str], one note per trace
    msg = client.messages.create(
        model="claude-opus-4-8", max_tokens=1500, thinking={"type": "adaptive"},
        messages=[{"role": "user", "content":
            "These are error-analysis notes from reading AI traces. Cluster them into 5-10 "
            "BINARY (observably yes/no) failure modes. For each give a short name, a one-line "
            "definition, and the note indices. Reply as JSON only:\n"
            '[{"name": "...", "definition": "...", "notes": [..]}]\n\n'
            + "\n".join(f"{i}: {c}" for i, c in enumerate(open_codes)))}])
    return json.loads(msg.content[-1].text)    # a DRAFT taxonomy — read it, fix it, make it yours

That suggest_taxonomy call is a draft generator, not an oracle: Claude returns candidate modes with definitions, and you read them against the traces, rename the vague ones, merge the duplicates, and split the ones hiding two failures. The taxonomy you ship is the one you stand behind.

See It: Build the Taxonomy

Here's the whole method in your hands. Below are 15 raw open codes from an error-analysis pass on a real-estate assistant. Do the axial coding: assign each note to a failure mode, and watch the messy pile resolve into a counted Pareto — with the biggest mode flagged as your first eval.

Interactive: 15 raw open-coding notes from error analysis (e.g. 'suggested a property over budget', 'invented an HOA fee', 'didn't call the availability API', 'returned unparseable JSON', 'I can't help with that for an in-scope question'). The reader axial-codes each note by clicking one of five failure modes (Constraint violation / Hallucination / Tool-retrieval miss / Format / Unhelpful refusal); a progress strip of 15 dots colours in as they go. A live Pareto bar chart of the taxonomy builds and re-sorts biggest-first, starring the #1 mode, and a 'top 3 cover X%' line appears. A '✦ Claude: suggest clusters' button auto-assigns all notes (the LLM-assistant) which the reader can override. The designed distribution: Constraint violation 6/15 (40%), Hallucination 4 (27%), Tool 3 (20%), Format 1, Refusal 1 — so the top 3 modes cover ~87% and Constraint Violation is the #1 eval to write first. Demonstrates open→axial coding, the Pareto, and fix-the-biggest-first prioritization.

Code all fifteen and the shape is unmistakable: five modes, but they're not equal — Constraint Violation alone is ~40%, and the top three cover ~87% of failures. That's the Pareto, and it's almost always there. You didn't just organize notes; you produced a ranked work order: the mode at the top is the bug to fix and the eval to write first, and the long tail (Format, Refusal at ~7% each) can wait. The count turned a feeling into a priority.

Count → Pareto → Prioritize

The third phase is where qualitative becomes quantitative. With the taxonomy fixed, you re-label every trace with its mode (now that the categories exist, this is fast) and count — a pivot table or three lines of code:

from collections import Counter
# Taxonomy fixed -> re-label EVERY trace with the final modes and COUNT. The pivot turns
# qualitative notes into a quantitative roadmap: it tells you which failure to fix FIRST.
def prioritize(labeled):                       # labeled: list[(trace_id, mode)]
    counts, total = Counter(m for _, m in labeled), len(labeled)
    cum = 0
    for mode, n in counts.most_common():       # sorted biggest-first
        cum += n
        print(f"{mode:<22} {n:3}  ({n/total:4.0%})   cumulative {cum/total:4.0%}")
    return counts.most_common()
# A handful of modes usually cause ~80% of failures (Pareto) -> write THOSE evals first (L173).
# Each mode is BINARY, so it's already an eval spec: "did the output violate a constraint? y/n".

The output is a Pareto chart of your failures, and in practice it's lopsided: “a simple pivot table revealed that just three issues accounted for most problems.” This is the single most useful artifact in evaluation, because it answers the question teams agonize over — “what do we work on?” — with data instead of opinion. Don't fix the failure that's the most embarrassing or the most interesting; fix the one that's most frequent (weighted by severity for high-stakes modes). The Pareto is also your antidote to the single-number trap from L159 (Eval Anti-Patterns (Vibes-Based Testing)): instead of one aggregate score that hides everything, you have a distribution that shows exactly where the product is breaking.

Iterate: Constant Comparison & Criteria Drift

Coding is not one-pass. Here's the part that surprises people: “after labeling everything, you go back and do it again from scratch — because you've shifted your own understanding of the evaluation criteria.” That's criteria drift (you met it in L168 (Writing Judge Prompts & Rubrics)), and it's not a bug — it's the method working. By trace 50 you understand “good” better than you did at trace 1, so your early labels deserve a second pass.

So you iterate (grounded theory's constant comparison): re-read, merge modes that turned out to be the same thing, split a mode that was secretly hiding two failures, rename for clarity, and re-count. You stop at theoretical saturation — when fresh traces stop producing new modes.

The common pitfalls to steer around:

Too many categories (over-splitting) — if you have 30 modes each with one example, you've coded symptoms, not modes. Lump them up.
Overlapping categories — if a failure plausibly fits three modes, your definitions aren't crisp; tighten them so each failure has one obvious home.
Premature top-down categories — imposing generic buckets before reading, which hides your real, product-specific failures.
An unrepresentative sample — code happy-path traces only and you'll build a taxonomy of failures you don't have.

The Payoff: Each Mode Becomes an Eval

Why go through all this? Because a finished failure taxonomy is the bridge from looking to measuring. A binary failure mode is, almost word-for-word, an eval specification: the mode “Constraint Violation” with the definition “did the output violate a stated constraint? y/n” is the spec for a judge or a check — you've already done the hard part (L168 (Writing Judge Prompts & Rubrics)) by writing the operational, binary criterion.

So the taxonomy hands the next lesson a prioritized backlog:

The Pareto order is your build order — write the eval for the #1 mode first (L173 (From Failures to Targeted Evals)).
For that top mode, assemble a gold-standard set — ~100 examples you've personally labeled pass/fail — to develop and calibrate the eval against (the calibration loop from L169 (Judge Biases & How to Calibrate)).
The whole thing plugs into the analyze → measure → improve loop (L174 (The Analyze → Measure → Improve Loop)): you analyzed (coded the taxonomy), you're about to measure (build the eval), then you improve and re-analyze.

That's the flywheel turning. You went from “the product feels off” to “40% of our failures are constraint violations, here's the eval, here's the gold set” — and that is the difference between guessing and engineering.

🧪 Try It Yourself

Turn last lesson's notes into a taxonomy — 30 minutes, and you'll have a real roadmap:

Gather your open codes. Take the free-text notes from your L171 trace read (or do a quick 20-trace pass now). One short label per trace, in a spreadsheet column.
Cluster them (axial coding). Read the column and group similar labels into 5–10 named modes. Stuck? Paste the notes into claude-opus-4-8 and ask it to “cluster these into 5–10 binary failure modes with definitions” — then edit its draft hard: rename, merge, split.
Make each mode binary. Rewrite every mode name as a yes/no question: “ignored a constraint? y/n,” “invented a fact? y/n.” If you can't, the mode is too vague.
Re-label and count. Go back through every trace, tag its mode, and count. Sort descending.
Read the Pareto. Which mode is #1? What % do the top three cover? That top mode is the eval you write next (next lesson) and almost certainly the bug you fix first.

If your top mode surprises you — “I had no idea constraint violations were 40% of our failures” — that surprise is exactly the payoff. You can't prioritize what you never counted.

Mental-Model Corrections

“I'll start with a standard list of failure categories.” Top-down blinds you to your product's real, weird failures. Open-code bottom-up and let the modes emerge from your data.
“Open coding = writing software.” No — it's qualitative labeling (grounded theory): a short descriptive note per trace. Stay descriptive (“over budget”), not categorical (“constraint violation”) until axial coding.
“Make the modes detailed and numerous.” Aim for 5–10 binary modes. Thirty modes with one example each are symptoms; lump them. Each mode should be observably yes/no.
“The taxonomy is the deliverable.” The count is. Re-label everything and build the Pareto — a taxonomy without frequencies can't prioritize.
“Fix the scariest / most interesting failure.” Fix the most frequent (weighted by severity). The Pareto, not your gut, sets the order.
“Code once and move on.” Iterate (constant comparison) — your criteria drift as you learn, so re-code, merge, split, until saturation.
“Let the LLM build the taxonomy.” Claude drafts the clusters; you read the traces, fix the categories, and own them. Assistant, not author.

Key Takeaways

Turn the pile of notes into a map with grounded theory: open coding (a short descriptive label per trace, bottom-up) → axial coding (cluster labels into 5–10 named, binary failure modes) → count.
Bottom-up, not top-down: let categories emerge from your data; a pre-built generic list hides your real, product-specific failures.
Count → Pareto → prioritize: re-label every trace and tally — a handful of modes cause most of the failures, so fix the biggest first (data, not opinion). It's the antidote to the single-number trap (L159).
Iterate: constant comparison — re-code as your criteria drift (L168), merge/split, stop at saturation. Avoid over-splitting, overlapping modes, premature categories, and unrepresentative samples.
Claude assists, you own it: the LLM proposes clusters; you read the traces and decide the taxonomy.
Each binary mode becomes an eval. The Pareto order is your build order, with a ~100-example pass/fail gold set for the top mode — straight into L173 (From Failures to Targeted Evals) and the analyze→measure→improve loop (L174 (The Analyze → Measure → Improve Loop)).