From Failures to Targeted Evals

Introduction

You did the hard, human work: you read the traces (L171 (Look at Your Data (The #1 Rule))) and turned the mess into a prioritized failure taxonomy (L172 (Open Coding & Failure Taxonomies)). Now comes the automation — but informed automation. This lesson turns the #1 failure mode on your Pareto chart into a specific, binary, validated eval, and then the next mode, and the next, until you have an eval suite that mirrors your actual failures.

Two ideas carry the whole lesson. First, specific beats generic: you don't build a vague “quality 1–5” judge — you build a scoped check for exactly the failure you found. Second, an eval is a classifier, so you have to validate it — an eval you haven't checked against human labels is just another untested hypothesis. This is where error analysis cashes out into engineering, and it's not a side quest: in practice, 60–80% of AI development time goes into error analysis and evaluation. This is the job.

An infographic titled 'From Failures to Targeted Evals' about turning the prioritized failure taxonomy from error analysis into specific, validated evals, one binary eval per failure mode in Pareto order. The first principle is specific over generic: a generic quality score wastes time and creates false confidence, so instead of one broad metric you write a scoped binary eval for exactly the failure mode you found, for example did the output violate a stated constraint, yes or no. The first decision for each mode is which grader to use: a code-based deterministic assertion or an LLM judge. Code assertions, such as a regular expression, JSON-schema validation, a contains check, an exact-value comparison, or an execution test, are right for verifiable structural properties like format, schema, length, and constraint checks; they are fast, free, fully reproducible, easy to audit, and need no model call. An LLM judge is right only for subjective qualities that rules cannot capture, like faithfulness, tone, helpfulness, and intent, where two expert reviewers given the same rubric would reliably agree. The rule of thumb is rule-based first, roughly a sixty-forty split, reserving the judge for the subjective remainder. Crucially the eval is itself a classifier of the failure mode, so you validate it against a human-labeled gold set of about a hundred pass-fail examples drawn from the tagged traces, and you measure not accuracy but the true positive rate and the true negative rate. The true positive rate is the fraction of real failures the eval catches; the true negative rate is the fraction of real passes it correctly passes. You use these instead of accuracy or precision because failure modes are rare and imbalanced, which makes accuracy misleadingly high and precision collapse, whereas the true positive and true negative rates are conditional on the true class and invariant to how rare the failure is. The bar is to iterate the eval until both the true positive rate and the true negative rate clear eighty percent, changing one thing at a time, tightening the check when it passes failures too easily and narrowing it when it false-alarms. The payoff is that the eval suite grows one targeted eval per top failure mode in Pareto order, becoming a regression gate, and since error analysis and evaluation absorb sixty to eighty percent of real AI development time, this is the work. The takeaway banner reads: turn each failure mode into one binary eval, code for verifiable properties and a judge for subjective ones, then validate it like a classifier until true positive rate and true negative rate both exceed eighty percent, because accuracy lies when failures are rare.

Specific Beats Generic

The instinct, once you decide to “add evals,” is to reach for a generic grader — “rate the overall quality 1–5,” “is this a good response?” Resist it. Generic evaluations waste time and create false confidence: a single fuzzy score tells you nothing actionable and (as you saw in L171) hides the failures that matter.

The alternative is targeted, scoped evals — one per failure mode you actually observed. Your taxonomy already named them in operational, binary terms (L172), so you've done the hard part: the mode “Constraint Violation — did the output violate a stated constraint? y/n” is an eval spec. Error analysis on your data surfaces failure modes no generic framework anticipates — a medical assistant that confuses adult vs. child dosages, a real-estate bot that suggests over-budget listings — and those deserve their own dedicated checks. So the move is: take the top mode on the Pareto, write a binary eval for precisely it, validate it, ship it, then descend to the next. Specific, in priority order — not generic, all at once.

The First Decision: Code or Judge?

For each failure mode, the first question is which grader — and it has a clear answer based on one property: is the failure verifiable or subjective?

Verifiable → a code assertion. If the failure is an observable property of the output — wrong format, invalid JSON, over a length limit, a price above the budget — write deterministic code: a regex, a json.loads, a contains, an ==, an execution test. It's fast, free, perfectly reproducible, easy to audit, and needs no model call (the cheap rung from L160 (Functional Correctness & Exact Match)).
Subjective → an LLM judge. If grading needs judgment — faithfulness, tone, helpfulness, intent — and “two experts with the same rubric would agree,” use a scoped binary judge (L166 (The Idea: Models Grading Models), L168 (Writing Judge Prompts & Rubrics)).

The rule of thumb: rule-based first, judge for the remainder — teams typically land near a 60/40 split. A code check beats a judge every single time it can do the job: no cost, no latency, no bias. Reserve the expensive, fallible judge for the failures that genuinely require reading meaning. Here's one targeted eval of each kind:

import json, re
from anthropic import Anthropic
client = Anthropic()

# VERIFIABLE mode -> a CODE assertion: fast, free, exact, no model call. One binary check.
def violates_budget(trace) -> bool:                     # "Constraint Violation"
    return any(p > trace["budget"] for p in trace["suggested_prices"])

def bad_json(trace) -> bool:                            # "Format" mode
    try: json.loads(trace["output"]); return False
    except json.JSONDecodeError: return True

# SUBJECTIVE mode -> a SCOPED, BINARY LLM judge (reserve it for what code can't check).
def unfaithful(trace) -> bool:                          # "Hallucination" mode
    r = client.messages.create(model="claude-opus-4-8", max_tokens=400, temperature=0,
        system='Reply JSON {"grounded": true|false}. grounded=false if ANY claim in the '
               'ANSWER is not directly supported by the SOURCE.',
        messages=[{"role": "user", "content": f'SOURCE:\n{trace["source"]}\n\nANSWER:\n{trace["output"]}'}])
    return not json.loads(r.content[-1].text)["grounded"]
# Rule of thumb: rule-based FIRST (~60%); spend a judge only on the subjective remainder.

Notice every eval returns a boolean — one failure mode, one yes/no. Binary checks are fast, deterministic to aggregate, and (next section) trivial to validate. The verifiable modes (violates_budget, bad_json) are pure code; only the genuinely subjective one (unfaithful) spends a Claude call.

Your Eval Is a Classifier — So Validate It

Here's the step teams skip and regret: an eval is a binary classifier of “does this trace exhibit failure mode X?” — and an unvalidated classifier is worthless. A budget check with a bug, or a faithfulness judge with a vague rubric, will hand you green checkmarks while the failures sail through. You have to measure the eval itself.

The procedure mirrors judge calibration (L169 (Judge Biases & How to Calibrate)):

Build a gold set — ~100 examples for this mode, each labeled pass/fail by a human (pull them from the traces you already tagged in L172; over-sample the failures so you have enough positives).
Run your eval on every example.
Compare the eval's verdict to the human label — a confusion matrix (caught / missed / false-alarm / passed; the structure from L163 (Task-Specific Metrics)).

The two errors have very different costs. A missed failure (the eval says pass, a human says fail) is the dangerous one — it's a bug your suite is blind to, giving false confidence. A false alarm (the eval cries fail on a good output) is annoying but visible. You need to quantify both — which is where the right metric matters.

See It: Build & Validate an Eval

Walk the whole pipeline. Pick a failure mode, choose a grader (code or judge), watch the binary eval render, and then validate it on a 50-example gold set — the confusion matrix and TPR/TNR appear. Try the wrong grader (a regex for tone, a judge for JSON) and watch the eval's TPR collapse while accuracy still looks deceptively fine.

Interactive: a 4-stage pipeline with arrow connectors. ① Pick a failure mode (Constraint violation / Format-JSON [verifiable] · Faithfulness / Tone [subjective]). ② Choose a grader: Code assertion or LLM judge. ③ The binary eval renders (e.g. 'any(price > budget ...)' for code, a yes/no rubric for judge). ④ It's validated on a 50-example human-labeled gold set (12 real failures): a 2x2 confusion matrix (caught TP / missed FN / false-alarm FP / passed TN) plus TPR, TNR, and accuracy. The teaching is in the routing: verifiable modes + code → TPR/TNR ~95% (ship); verifiable + judge → ~80% (works but overkill); subjective + judge → ~87% (ship); subjective + code (regex for tone/hallucination) → TPR ~25-30% (it MISSES most failures) even though accuracy still reads ~70%+ — which is exactly why you gate on TPR & TNR, not accuracy, since failures are rare. Lets the reader build and validate 8 mode×grader combinations and see why specific grader choice + validation matter.

Flip through the combinations and the lesson is unmistakable. Constraint Violation + code validates near-perfect; Faithfulness + code (a regex pretending to detect hallucination) has a TPR around 30% — it misses two-thirds of real failures — yet its accuracy still reads ~70%+, because most outputs are fine and a lazy “pass” is right by default. An eval you judged by accuracy would have shipped blind. That gap is the whole reason for the next idea.

Use TPR & TNR, Not Accuracy

Failure modes are rare — most outputs are fine — and that breaks the obvious metrics. With an 8%-prevalence failure, an eval that does nothing (always says “pass”) scores 92% accuracy, and precision swings wildly on a handful of positives. Accuracy and precision lie when classes are imbalanced.

The metrics that don't lie are TPR and TNR:

TPR (True Positive Rate / recall / sensitivity) = of the real failures, what fraction did the eval catch? TP / (TP + FN).
TNR (True Negative Rate / specificity) = of the real passes, what fraction did the eval correctly pass? TN / (TN + FP).

The key property: TPR and TNR are conditional on the true class, so they're invariant to how rare the failure is. A lazy always-pass eval has 92% accuracy but a TPR of 0% — and TPR exposes it instantly. The bar, from the practitioners who do this for a living: iterate the eval until TPR and TNR both clear 80%.

# Your eval IS a classifier of one failure mode -> validate it on a HUMAN-labeled gold set
# before you trust it. Use TPR & TNR, not accuracy: failure modes are RARE, so accuracy and
# precision are misleading; TPR/TNR are conditional on the true class -> invariant to prevalence.
def validate(eval_fn, gold):                  # gold: [(trace, human_fail: bool), ...]
    tp = fn = fp = tn = 0
    for trace, human_fail in gold:
        flag = eval_fn(trace)                  # does the eval say this trace FAILS the mode?
        if   human_fail and flag:       tp += 1
        elif human_fail and not flag:   fn += 1     # MISSED a real failure  (kills TPR)
        elif not human_fail and flag:   fp += 1     # false alarm            (kills TNR)
        else:                           tn += 1
    tpr = tp / (tp + fn)        # of REAL failures, the fraction the eval catches
    tnr = tn / (tn + fp)        # of REAL passes,   the fraction it correctly passes
    print(f"TPR={tpr:.0%}  TNR={tnr:.0%}  ->  {'ship' if tpr>=.8 and tnr>=.8 else 'iterate'}")
    return tpr, tnr
# Need 100+ labeled examples. If either is < 80%, fix the eval — ONE change — and re-validate.

That's the gate: a targeted eval isn't done when the code runs — it's done when, on 100+ human-labeled examples, it catches ≥80% of real failures (TPR) and correctly passes ≥80% of good outputs (TNR). Anything less and your “eval” is just a comforting green light. (And remember the number is a noisy estimate — put a CI on it, L165 (Statistical Rigor: Sample Size & Confidence).)

Iterate the Eval, Then Grow the Suite

Validation rarely passes on the first try — and the fix is a disciplined loop, not a rewrite:

Low TPR (missing failures)? Your check is too lax — broaden the regex, sharpen the rubric, add the edge case you missed. Look at the false negatives specifically.
Low TNR (false alarms)? Your check is too aggressive — tighten the condition, add the legitimate exception. Look at the false positives.
Change ONE thing at a time, then re-validate. (And if cases “pass too easily,” tighten the assertion until the real failures actually fail it.)

This is the same calibration loop as L169 (Judge Biases & How to Calibrate) — you're aligning the eval to your humans. Once it clears the 80/80 bar, add it to the suite and move to the next Pareto mode. The suite grows one targeted eval at a time, in priority order, until it covers the failures that actually happen — and that suite becomes your regression gate (L175 (Catching Regressions Before They Ship)) and the “measure” step of the loop (L174 (The Analyze → Measure → Improve Loop)). You started with “the product feels off”; you now have “here are our 6 failure modes, each with a validated eval at >80% TPR/TNR, run on every change.” That's engineering.

🧪 Try It Yourself

Build and validate one real eval — the top mode from your L172 taxonomy — in under an hour:

Pick the #1 mode. Take the most frequent failure mode from your Pareto. Write it as a yes/no question.
Route it. Is it verifiable (a checkable property → write a code assertion: regex, json.loads, a comparison) or subjective (→ a scoped binary claude-opus-4-8 judge with a one-line rubric)? Default to code if you possibly can.
Build a 30–100 example gold set. Pull tagged traces, and label each pass/fail yourself. Over-sample the failures so you have ≥15 positives.
Validate. Run your eval, build the confusion matrix, compute TPR and TNR. Are both ≥80%?
Iterate once. Read the misses (false negatives) — fix the one thing causing them — and re-run. Watch TPR climb.

When your eval clears 80/80, you've converted a vibe (“it sometimes ignores constraints”) into an instrument (“our constraint eval catches 91% of violations, runs on every PR”). Do it for the next mode tomorrow.

Mental-Model Corrections

“Add a generic ‘quality 1–5’ eval.” Generic scores waste effort and create false confidence. Build a specific, binary eval per observed failure mode, in Pareto order.
“Use an LLM judge for everything.” Code first. If the failure is a verifiable property (format, schema, constraint), a regex/json.loads/== is faster, free, and more reliable than a judge. Reserve the judge for the subjective ~40%.
“The eval works because the code runs.” An eval is a classifier you must validate against human labels — an unvalidated eval can be confidently, silently wrong.
“Judge the eval by accuracy.” Failures are rare, so accuracy (and precision) lie — an always-pass eval can hit 92% accuracy with 0% TPR. Use TPR & TNR, which are invariant to prevalence.
“80% agreement is fine.” The bar is both TPR and TNR ≥ 80% — high TPR with low TNR floods you with false alarms; high TNR with low TPR misses the failures.
“Rewrite the eval when it fails validation.” Iterate — one change at a time, reading the false negatives (raise TPR) or false positives (raise TNR) — like calibrating a judge (L169).

Key Takeaways

Turn each failure mode into one targeted, binary eval (specific > generic), built in Pareto order from your taxonomy (L172 (Open Coding & Failure Taxonomies)).
Choose the grader by the failure's nature: verifiable → a code assertion (regex/schema/==/exec — fast, free, exact; L160); subjective → a scoped binary LLM judge (L166, L168). Rule-based first, ~60/40.
An eval is a classifier — validate it against a ~100-example human-labeled gold set with a confusion matrix (caught / missed / false-alarm / passed). A missed failure is the dangerous error.
Use TPR & TNR, not accuracy: failures are rare, so accuracy/precision mislead; TPR (catch real failures) and TNR (pass real passes) are invariant to prevalence. Iterate until both ≥ 80% (one change at a time).
Grow the suite one validated eval per top mode — it becomes your regression gate (L175 (Catching Regressions Before They Ship)) and the measure step of the loop (L174 (The Analyze → Measure → Improve Loop)). Error analysis + evals are 60–80% of AI dev time — this is the work.