Task-Specific Metrics (Classification, Extraction, Summarization)

Introduction

You now have a toolbox of metrics — exact match, similarity, perplexity — and the most important meta-skill: knowing that no single metric is universal. The right one depends entirely on the shape of the task. Grading a yes/no classifier like an open-ended summary (or vice-versa) gives you a confident, meaningless number.

This lesson walks the three most common task shapes and the metrics built for each: classification (did it pick the right label?), extraction (did it pull the right fields?), and summarization (is the summary good and true?). Along the way you'll meet the single most dangerous metric in all of evaluation — accuracy — and learn exactly why it lies, and what to use instead. Master this and you'll reach for the metric that measures what actually matters for the task in front of you.

An infographic titled 'Task-Specific Metrics' showing that there is no universal evaluation metric and the skill is matching the metric to the task shape, illustrated across three common shapes. For CLASSIFICATION, every prediction reduces to a confusion matrix of four counts: true positives and true negatives that are correct, and false positives and false negatives that are wrong; from these come accuracy equals true positives plus true negatives over all, precision equals true positives over true positives plus false positives, which asks when I flag positive am I right, recall equals true positives over true positives plus false negatives, which asks of all the real positives did I catch them, and F1 the harmonic mean of precision and recall. A warning panel says accuracy lies on imbalanced data, because a lazy model that always predicts the majority class can score ninety-nine percent accuracy while catching zero of the rare positives, so recall is zero; the fix is precision, recall, F1, or a precision-recall curve. Precision and recall trade off and you tune the threshold: favor precision when false alarms are costly, like spam filtering or fraud, and favor recall when a miss is costly, like medical diagnosis or security screening. For EXTRACTION of structured fields, entities, or tool arguments, score the same precision, recall, and F1 but at the field level, of the fields I extracted how many are correct and of the fields that should be there how many did I find, choosing between strict exact match and a more lenient partial match. For SUMMARIZATION, n-gram overlap like ROUGE is not enough, so grade multiple dimensions led by faithfulness or factual consistency, meaning every claim is supported by the source, since a fluent summary that invents facts is worthless, plus coverage of the key points, conciseness, and fluency; faithfulness is a semantic check that overlap cannot do, so it is measured by natural-language-inference models or, increasingly, an LLM judge. A band gives the unifying idea: match the metric to the task shape and layer them in production as an evaluation pyramid, a deterministic floor on every request, a cheap fine-tuned classifier on every output, and an LLM judge on a sample for open-ended rubrics like faithfulness and helpfulness. The takeaway banner reads: there is no one metric, classification wants precision, recall and F1 with an eye on imbalance and error costs, extraction wants field-level F1, and summarization wants faithfulness first, so pick by task shape and combine by cost.

Classification: the Confusion Matrix

A huge share of AI work is classification: sentiment, intent routing, a guardrail labeling a message safe or unsafe, a triage bot tagging a ticket urgent or not. Every one of them reduces to the same object — the confusion matrix — which sorts predictions into four buckets:

TP (true positive) — flagged positive, and it was. ✓
FP (false positive) — flagged positive, but it wasn't. (a false alarm)
FN (false negative) — said negative, but it was positive. (a miss)
TN (true negative) — said negative, and it was. ✓

From those four counts come the metrics that actually matter — and notice that precision and recall ask genuinely different questions:

# Every classification reduces to a confusion matrix. From its four counts:
def metrics(tp, fp, fn, tn):
    accuracy  = (tp + tn) / (tp + fp + fn + tn)
    precision = tp / (tp + fp)          # "when I flag positive, am I right?"
    recall    = tp / (tp + fn)          # "of all the real positives, did I catch them?"
    f1        = 2 * precision * recall / (precision + recall)   # harmonic mean of P & R
    return dict(acc=accuracy, prec=precision, rec=recall, f1=f1)

# A guardrail flagging UNSAFE messages -- only 8 of 1000 are unsafe (wildly imbalanced):
metrics(tp=5, fp=2, fn=3, tn=990)       # -> acc 99.5% . prec 71% . rec 63% . f1 67%

# The trap: a lazy model that ALWAYS predicts "safe" scores acc = 992/1000 = 99.2% --
# yet recall is 0%: it catches NONE of the unsafe messages. Accuracy lies on imbalance.

Precision = TP/(TP+FP) — “when I flag positive, how often am I right?” (it punishes false alarms). Recall = TP/(TP+FN) — “of all the real positives, how many did I catch?” (it punishes misses). F1 is their harmonic mean — a single number that's only high when both are. And accuracy — the one everyone reaches for first — is about to betray you.

Why Accuracy Lies (the Imbalance Trap)

Accuracy — “what fraction did I get right?” — feels like the obvious metric. It's a trap on imbalanced data, which is most real data: most messages are safe, most tickets aren't urgent, most transactions aren't fraud.

Here's the betrayal in one line: if only 8 of 1000 messages are unsafe, a lazy model that always predicts “safe” is right 992 / 1000 = 99.2% of the time. A 99.2% accuracy! And it has caught zero unsafe messages — its recall is 0%. The metric looks brilliant; the model is worthless.

This is why, on imbalanced data, accuracy is almost never the right primary metric. Reach instead for precision, recall, F1, or the precision-recall curve (PR-AUC) — all of which keep their eye on the rare positive class that accuracy happily ignores. The interactive below lets you watch accuracy stay high while recall collapses.

Precision vs Recall: a Tradeoff You Tune

Precision and recall are in tension — push one up and the other usually drops — and the lever between them is the decision threshold. Raise the threshold (only flag when very confident) and you make fewer positive predictions: precision rises (fewer false alarms), recall falls (more misses). Lower it and the reverse happens.

There's no universally “right” setting — it depends on which error is more expensive for your problem:

Favor precision when false positives are costly. A spam filter dumping a real invoice into junk, or a fraud system declining a legitimate customer's card — those false alarms do real damage, so you'd rather miss some spam than block good mail.
Favor recall when false negatives are costly. A cancer screen that misses a tumor, a security scan that misses a weapon, a moderation filter that lets abuse through — here a miss is catastrophic, so you accept more false alarms to catch everything.
Use F1 (or tune the threshold to a target) when you genuinely need to balance both.

For multi-class problems, watch the averaging: macro-F1 (average the per-class F1s) treats every class equally and exposes weakness on rare classes, while micro-F1 is dominated by the common ones. On imbalanced multi-class, prefer macro.

See It: Drag the Threshold

Time to feel the tradeoff in your hands. Below, a classifier flags urgent support tickets — and only 8 of 40 are truly urgent. Drag the decision threshold and watch the confusion matrix and all four metrics move together — and watch accuracy lie while recall quietly collapses.

Interactive: an urgent-ticket classifier on an imbalanced set (8 of 40 urgent). Drag the decision threshold and the confusion matrix (TP/FP/FN/TN) and the four metrics (accuracy, precision, recall, F1) recompute live. Crank the threshold up and accuracy stays high (~80%) while recall collapses toward zero — the imbalance trap — and a dynamic note explains the precision-recall tradeoff at each setting.

Did you catch the lie? At a high threshold the model flags almost nothing, accuracy looks fine (~80%) because most tickets really aren't urgent — but recall craters, and you're missing the very tickets the system exists to catch. The threshold is a dial between false alarms and misses; you set it by the cost of each.

Extraction: Field-Level Precision & Recall

The second shape is extraction — pulling structured items out of text: named entities, key-value pairs, the fields of a JSON object, the arguments of a tool call. The good news: it's classification in disguise, so the same machinery applies — just at the level of individual fields rather than whole documents.

Score each extracted item as TP/FP/FN and compute precision (of the fields I pulled out, how many are correct?), recall (of the fields that should be there, how many did I find?), and their F1. A model that extracts ten fields and gets seven right but misses three has a precision and recall story worth telling separately.

The one real decision is exact vs partial match. Does "Jan 5, 2026" count as a match for "2026-01-05"? Is "Dr. Maria Santos" a hit for "Maria Santos"? Exact match is strict and unambiguous (great for IDs and enums); partial match awards credit for overlapping spans (kinder for names and free-text). Pick the strictness your use case can tolerate — and when the boundary is fuzzy, a normalized exact match (from the functional-correctness lesson) usually splits the difference.

Summarization: Beyond Overlap — Faithfulness First

The third shape is open-ended generation, and summarization is the canonical case. We already saw (in Similarity Metrics) why ROUGE alone is not enough — it counts n-gram overlap, so it rewards copying the reference's words and is blind to whether the summary is actually true. A good summary has to be graded on several dimensions at once, and they're not equally important:

Faithfulness / factual consistency — the non-negotiable one. Is every claim in the summary supported by the source? A fluent, well-organized summary that invents a fact is worse than useless. Because this is a semantic check — overlap can't see a flipped or fabricated claim — it's measured with NLI-based methods (FactCC, SummaC) or, increasingly, an LLM judge:

# Summaries need more than overlap. FAITHFULNESS is #1: is every claim in the summary
# supported by the source? That's a SEMANTIC check -- ask a judge (Section 3):
from anthropic import Anthropic
client = Anthropic()

def is_faithful(summary: str, source: str) -> bool:
    r = client.messages.create(
        model="claude-opus-4-8", max_tokens=10,
        system="Reply FAITHFUL only if EVERY claim in the summary is supported by the "
               "source; otherwise reply UNFAITHFUL.",
        messages=[{"role": "user", "content": f"SOURCE:\n{source}\n\nSUMMARY:\n{summary}"}])
    return r.content[0].text.strip().startswith("FAITHFUL")

# ROUGE can't see a flipped or invented fact; this can. (Then add coverage & conciseness.)

Coverage / relevance — did it capture the key points, not trivia?
Conciseness — is it tight, or padded with fluff?
Fluency / coherence — does it read well and hang together?

Notice the pattern: the moment a task is genuinely open-ended, deterministic metrics run out, and the meaningful dimensions (especially faithfulness) are best measured by a judge — which is exactly where the next section goes. For summaries, faithfulness is the floor: get that wrong and the other three don't matter.

The Unifying Idea: Match the Metric — and Layer Them

Step back and the whole lesson is one principle: match the metric to the task's shape.

Classification (labels, routing, guardrails) → precision / recall / F1, with a hard eye on imbalance and the cost of each error. Never accuracy alone.
Extraction (fields, entities, tool args) → the same, at the field level, choosing exact vs partial match.
Summarization / open-ended (and RAG answers, chat, writing) → faithfulness first, then coverage, conciseness, fluency — mostly via a judge.

And in production you don't pick one — you layer them into an evaluation pyramid, cheapest first: a deterministic floor on every request (schema/format checks, microseconds, free), a cheap fine-tuned classifier on every output (sharp targets like safety, milliseconds, a fraction of a cent), and an LLM judge on a sample for the open-ended rubrics (faithfulness, helpfulness). Pick the metric by task shape; combine the layers by cost. That's how mature teams evaluate without going broke — or blind.

🧪 Try It Yourself

Take one task your system performs and put it through the lens:

Name its shape. Is it classification (a label), extraction (fields), or open-ended (a summary, an answer, prose)? That alone tells you which metric family to reach for.
If it's a classifier, sketch its confusion matrix on ~20 real examples. Compute precision and recall by hand. Then ask the decisive question: which error hurts more here — a false alarm or a miss? That answer sets your threshold, not a default 0.5.
If it's extraction, list the fields and decide exact vs partial match for each (an ID is exact; a free-text name is partial).
If it's open-ended, write a one-sentence faithfulness check you could hand to a judge: “PASS only if every claim is supported by the source.”

If your current metric is accuracy on imbalanced data — or ROUGE on a summary — you've just found a number that's been quietly lying to you. Swap it for the one that fits the shape.

Mental-Model Corrections

“Accuracy tells me how good my classifier is.” On imbalanced data it lies — always-predict-majority can hit 99% while catching zero positives (recall 0%). Use precision, recall, F1, or PR-AUC.
“Always maximize F1.” Only when both errors cost the same. Favor precision when false alarms hurt (spam, fraud); favor recall when misses hurt (medical, security). Tune the threshold to the cost.
“ROUGE measures summary quality.” It measures word overlap. A summary can have high ROUGE and still invent facts. Grade faithfulness first (a semantic check), then coverage and conciseness.
“Extraction needs exact match.” Sometimes — for IDs and enums. For names, dates, and free text, partial / normalized match is often fairer. Score at the field level with P/R/F1.
“Pick one metric for the whole system.” No — match each metric to each task's shape, and layer cheap-to-expensive (deterministic → classifier → judge). One number rarely covers a real system.
“The default 0.5 threshold is fine.” It's arbitrary. The right threshold is wherever your precision-recall tradeoff lands given your error costs.

Key Takeaways

No universal metric — match it to the task shape. Classification → precision/recall/F1; extraction → field-level P/R/F1; summarization/open-ended → faithfulness + coverage + conciseness.
Every classification is a confusion matrix. Precision = TP/(TP+FP) (false-alarm-averse), recall = TP/(TP+FN) (miss-averse), F1 = their harmonic mean.
Accuracy lies on imbalanced data — always-predict-majority scores ~99% with 0% recall. Use precision/recall/F1/PR-AUC, and macro-averaging for imbalanced multi-class.
Precision vs recall is a tradeoff you tune with the threshold, set by the cost of the error: precision when false positives hurt (spam, fraud), recall when false negatives hurt (medical, security).
Extraction is classification at the field level — score P/R/F1 over fields/entities/tool-args, choosing exact vs partial match.
Summaries need faithfulness first (every claim supported by the source — a semantic check via NLI or a judge), beyond ROUGE's word overlap. Match by shape; layer by cost (deterministic → classifier → judge).