Functional Correctness & Exact Match

Introduction

You can build an eval set (Building Your First Eval Set) and dodge the traps (Eval Anti-Patterns). Now Section 2 gets specific: how do you actually grade an output? There's a whole spectrum of methods, and the golden rule is to start at the cheap end.

This lesson is that cheap end — the deterministic, code-based graders: functional correctness and exact match. When your output is verifiable, you don't need a fuzzy similarity score or an expensive LLM judge; you need a == or a unit test. These graders are fast, free, perfectly reproducible, and run in CI like any other test. Reach for them first, every time you can — and know exactly where they stop working.

An infographic titled 'Functional Correctness & Exact Match' about the cheapest, most reliable family of LLM graders: the deterministic, code-based ones, which need no LLM judge. On the left, FUNCTIONAL CORRECTNESS is shown as the gold standard: instead of comparing generated code to a reference string, you RUN it in a sandbox against unit tests, and exit code zero means every assertion held, so it actually works; this is execution-based evaluation, the basis of HumanEval, the pass-at-k metric, and SWE-bench, and the point is that it tests the thing itself rather than a proxy. On the right, EXACT MATCH is the simplest grader, checking whether the output equals the reference, ideal for single-canonical-answer tasks like classification labels, IDs, yes or no, and extracted fields; but raw equality is brittle, so the string thirty Days with a capital D and a period fails against thirty days until you NORMALIZE both sides with the standard recipe of lowercasing, stripping punctuation, collapsing whitespace, and removing articles, while warning that over-normalization invites false positives, and that the phrase thirty spelled out never matches the digits, the ceiling where string comparison fails. A central band shows the VERIFIABILITY SPECTRUM: verifiable tasks such as code, math, structured data, and classification have an unambiguous ground truth and should be graded with cheap, fast, objective deterministic checks, while unverifiable tasks such as summaries and chat have no single right answer, where exact match is a category error and you need similarity metrics or an LLM judge, covered in the next lessons. The deterministic toolkit beyond equality is listed: JSON schema validation, regular expressions, substring contains, numeric tolerance, and set or F1 match for tool calls. The guiding principle is to prefer deterministic graders where possible, an LLM judge where necessary, and humans judiciously, always reaching for the cheapest grader that fits. The takeaway banner reads: if the output is verifiable, grade it by running it or matching it, the fastest, most trustworthy signal you can get; reserve judges for what truly cannot be checked.

The Grader Hierarchy: Prefer Deterministic

Before any specific metric, internalize the ordering. Anthropic's eval guidance puts it crisply: choose deterministic graders where possible, LLM graders where necessary, and human graders judiciously. It's a ladder of cost and reliability:

Deterministic / code-based — fast, free, objective, 100% reproducible. Same input → same verdict, forever.
LLM-as-judge — flexible enough for fuzzy outputs, but slower, costs tokens, and is itself biased and non-deterministic (Section 3).
Human — the gold standard for nuance, but slow and expensive; use it to calibrate the others.

The discipline is simple: always reach for the cheapest grader that actually fits your output. A deterministic check you can run ten thousand times in CI for free beats a clever LLM judge you have to pay for and second-guess. This lesson is the cheapest rung — use it wherever your output lets you.

Functional Correctness: Does It Actually Work?

The strongest grader of all doesn't look at your output at all — it runs it. For anything executable — code, SQL, a formula, a tool call — don't compare the generated text to a reference; execute it against tests and check whether it works.

This is execution-based evaluation, and it's the foundation of every serious code benchmark: HumanEval (164 problems graded by running unit tests), the pass@k metric (does at least one of k samples pass?), and SWE-bench, which grades models on real GitHub issues by running the repo's test suite — where scores climbed from ~40% to over 80% in a single year. The magic is that it tests the thing itself, not a proxy: two completely different-looking functions both pass if they both compute the right answer.

import anthropic, subprocess
client = anthropic.Anthropic()

# FUNCTIONAL CORRECTNESS: don't compare the code to a reference string -- RUN it.
TESTS = "assert is_palindrome('racecar')\nassert not is_palindrome('hello')"

def passes_tests(task: str) -> bool:
    code = client.messages.create(
        model="claude-opus-4-8", max_tokens=400,
        messages=[{"role": "user", "content": f"Write a Python function. {task} Code only."}]
    ).content[0].text
    program = code + "\n" + TESTS
    result = subprocess.run(["python", "-c", program],          # run it in a SANDBOX
                            capture_output=True, timeout=5)
    return result.returncode == 0          # exit 0 = every assert held = it WORKS

passes_tests("def is_palindrome(s): True iff s reads the same backwards")
# -> True   graded by EXECUTION, not by how the code looks

returncode == 0 is the entire grader: every assertion held, so the code is functionally correct — regardless of style, variable names, or how it's phrased. (Always run untrusted, model-generated code in a sandbox with a timeout — more on that in the agents material.) When you can execute the output, this is the most trustworthy signal in all of evaluation.

Exact Match: the Simplest Grader

When the output isn't executable but has a single canonical answer, the simplest deterministic grader is exact match: output == reference. It's perfect for:

Classification — is the predicted label exactly the gold label?
Multiple choice / yes-no — "B" == "B".
IDs, codes, structured fields — order numbers, SKUs, dates, enums.
Extraction — did you pull out exactly "30 days"?

It's unbeatable when it fits: instant, free, and utterly unambiguous. There's no judge to bias, no metric to game. The catch is in that word exactly — raw string equality is famously brittle.

Normalized Exact Match: Taming the Brittleness

Raw == fails for silly reasons. "30 Days." ≠ "30 days" because of one capital letter and a period — even though they're obviously the same answer. As we saw in Why Evaluating LLMs Is Uniquely Hard, strict equality marks good answers wrong over pure surface form.

The fix is normalization: clean both sides with a deterministic procedure before comparing. The standard recipe (used by datasets like SQuAD) is lowercase, strip punctuation, collapse whitespace, and remove articles — plus canonicalizing numbers or units when relevant:

import re

# EXACT MATCH, done right: NORMALIZE both sides before comparing (the SQuAD recipe).
def normalize(s: str) -> str:
    s = s.lower().strip()
    s = re.sub(r"[.,!?;:'\"()]", "", s)       # strip punctuation
    s = re.sub(r"\b(a|an|the)\b", " ", s)      # drop articles
    return re.sub(r"\s+", " ", s).strip()      # collapse whitespace

def exact_match(pred: str, gold: str) -> bool:
    return normalize(pred) == normalize(gold)

exact_match("30 Days.",               "30 days")   # -> True   (formatting forgiven)
exact_match("The answer is 30 days.", "30 days")   # -> False  (extra words != gold)
# Normalize enough to forgive FORMATTING -- but no more, or you invite false positives.

Normalize just enough to forgive formatting — and no more. Over-normalize (e.g., strip so much that "$30" and "30%" collapse together) and you swing the other way into false positives, marking wrong answers correct. Match the strictness to the answer type: a date field tolerates whitespace; a currency value does not.

See It: Build a Robust Exact-Match Grader

Watch the trade-off live. Below, six model outputs are checked against the reference "30 days" — five genuinely mean 30 days, one doesn't. Start with raw ==, then toggle normalizations and watch the grader's accuracy against ground truth climb — and find the case it can never catch.

Interactive: grade six outputs against the reference '30 days' by toggling normalizations (lowercase, collapse whitespace, strip punctuation, allow substring). Raw == is brittle (it wrongly fails good answers); sensible normalization fixes most; 'allow substring' is looser and risks false positives; and 'thirty days' never matches — the ceiling of string comparison. A live accuracy score shows how well your grader agrees with the truth.

The lesson in your hands: a few sensible normalizations turn brittle == into a robust grader — but "thirty days" exposes the ceiling. No string rule understands that thirty means 30. For that kind of meaning-level equivalence you need a semantic metric or a judge — which is exactly where the next lessons go.

The Deterministic Toolkit Beyond ==

Exact match is the headline, but the deterministic family is bigger — and none of these need an LLM. Pick the check that matches your output's shape:

JSON-schema validation — does the output parse into the expected schema, with the right fields and types? Free, instant, and the backbone of any structured-output pipeline.
Regex / pattern match — is it a valid email, a date in YYYY-MM-DD, a phone number? Does the required citation marker exist?
Substring / contains — is the key fact present somewhere in a longer answer? (Looser — use with care, as you just saw.)
Numeric tolerance — is the number within ±ε of the target? (For floats, money, measurements.)
Set / F1 match — for multi-label extraction or tool-call correctness: did the agent call the right set of tools? Compare predicted vs expected sets with precision/recall/F1.

All of them are fast, cheap, and deterministic — they gate a pull request or block a deploy exactly like a unit-test suite. The skill is choosing the right strictness: tight enough to catch real errors, loose enough not to punish valid formatting.

The Verifiability Spectrum: When This Works (and When It Doesn't)

Step back and the whole lesson reduces to one question you should ask of every output: is it verifiable?

Verifiable tasks have an unambiguous ground truth — code (run the tests), math (check the number), structured data (validate the schema), classification (match the label), tool selection (compare the set). Here, deterministic graders are not just good enough — they're ideal: cheaper, faster, and more trustworthy than any judge. This is the same property that makes these tasks suitable for verifiable rewards in model training.
Unverifiable tasks have no single right answer — summaries, conversations, creative writing, explanations. Here, exact match is a category error (forcing one valid answer to be the answer), and you must move up the ladder to similarity metrics (next lesson) or an LLM-as-judge (Section 3).

So the rule that ties it together: if the output is verifiable, grade it deterministically — run it or match it. Reserve the slow, fuzzy, expensive graders for the outputs that genuinely can't be checked any other way. Most teams reach for an LLM judge far too early, on tasks a three-line == would have nailed for free.

🧪 Try It Yourself

Take one output type from your system and place it on the spectrum:

Ask: is it verifiable? Does it have a single correct answer, a schema, or a test you could run? If yes, you're in deterministic territory — don't reach for a judge.
Write the cheapest grader that fits. Executable → run it against an assertion. Structured → validate the schema. Single answer → exact_match with the normalize recipe. A number → numeric tolerance.
Try to break it. Feed it a correct answer in a slightly different format — does your grader wrongly fail it? Add the minimal normalization. Then feed it a wrong answer that's superficially close — does it wrongly pass? Tighten up.
Find the ceiling. If you hit an output where no rule captures "same meaning" (like "thirty" vs 30, or two valid summaries), write it down — that's a case for the next lessons' metrics, not for ==.

The goal: grade as much as you can for free, and know precisely which outputs force you up the ladder.

Mental-Model Corrections

“Exact match is too crude to be useful.” Backwards — when the output is verifiable, it's the best grader: instant, free, unbiased, ungameable. Crude is a feature.
“I need an LLM judge to evaluate anything real.” Most teams over-reach for judges. Prefer deterministic graders; a unit test or == is cheaper and more trustworthy whenever it fits.
“Raw == is fine.” It's brittle — one capital letter or trailing period fails a correct answer. Normalize (lowercase, punctuation, whitespace, articles) before comparing.
“So normalize as hard as possible.” No — over-normalization causes false positives, marking wrong answers right. Match the strictness to the answer type.
“Functional correctness is only for code.” It's for anything executable or checkable — SQL, formulas, tool calls, structured outputs. If you can run or validate it, do that instead of comparing strings.
“Every output needs the same grader.” No — first ask “is this verifiable?” Verifiable → deterministic; unverifiable → similarity or judge. Picking the grader is half the job.

Key Takeaways

Start at the cheap end. Prefer deterministic, code-based graders — fast, free, objective, perfectly reproducible, CI-friendly — and only climb to LLM judges (Section 3) or humans when the output forces you to.
Functional correctness is the gold standard for executable outputs: don't compare to a reference, run it against tests (HumanEval, pass@k, SWE-bench). It grades the thing itself, not a proxy.
Exact match is unbeatable for single-answer tasks (labels, IDs, fields) — but raw == is brittle. Use normalized exact match (lowercase, strip punctuation, collapse whitespace, drop articles), and don't over-normalize into false positives.
The toolkit is bigger than ==: JSON-schema validation, regex, contains, numeric tolerance, and set/F1 (tool-call correctness) — all deterministic, all LLM-free. Choose the right strictness for the output type.
Ask “is it verifiable?” of every output. Verifiable (code, math, structured, labels) → grade deterministically. Unverifiable (summaries, chat) → exact match is a category error; move up to similarity metrics or a judge.
The ceiling is meaning: no string rule knows "thirty" = 30. When you hit semantic equivalence, that's the boundary of this lesson — and the start of the next ones.