Why Evaluating LLMs Is Uniquely Hard

Introduction

The previous lesson, The Eval-Driven Development Mindset, made the case that you must measure. The obvious next thought is: “fine — I'll just write tests, like I do for normal code.” And then you hit a wall. Evaluating an LLM breaks almost every intuition you've built from a career of software testing.

This lesson is about why — and it matters, because if you don't respect how this is hard, you'll reach for the wrong tools (an assertEqual here, a public benchmark there) and quietly fool yourself. Understanding the shape of the difficulty is what lets you pick the right tools in the rest of this container. Spoiler: the hardness isn't a flaw to route around. It's exactly why eval skill is so valuable.

An infographic titled 'Why Evaluating LLMs Is Uniquely Hard' contrasting traditional software testing with LLM evaluation. On the left, labeled TRADITIONAL CODE, a single assertion reads assert add of 2 and 2 equals 4, with a green check, and three captions: deterministic — same input, same output; one correct answer; cheap to check. On the right, labeled AN LLM ANSWER, the call summarize(ticket) fans out to three different but equally valid summaries, and an attempt to assert that the output equals a single reference fails with a red cross even though the output was good, because there is no single reference; below it, five human graders score the same answer and disagree — some pass, some borderline, one fail — with the caption no single right answer, and even experts disagree. Across the center sit the Three Gulfs from Hamel Husain and Shreya Shankar, drawn as three gaps: the Gulf of Comprehension between you and your data (you can't read ten thousand traces); the Gulf of Specification between your intent and the prompt (you can't precisely say what you want); and the Gulf of Generalization between your few examples and the long tail (it works but isn't robust). A band warns that public benchmarks won't save you, listing Goodhart's law (when a measure becomes a target it stops measuring), saturation (top models cluster at the ceiling), and contamination (test questions like MMLU found verbatim in the Common Crawl training data, so the score is memorization, not reasoning) — benchmarks are opinion polls, not thermometers. Three cards summarize: first, the output space is effectively infinite and non-deterministic, so exact-match testing simply does not apply; second, the Three Gulfs are why it is hard — comprehension, specification, and generalization; third, the difficulty is good news, because eval skill is exactly the moat — most teams give up and ship on vibes, and the ones who do the hard thing win. A takeaway banner reads: there is often no single right answer and even experts disagree, so grading an LLM is not reading an answer key — you must design what good means, and that design problem is the hard, valuable part of AI engineering.

Your Software-Testing Intuitions Don't Transfer

Normal code is a joy to test. assert add(2, 2) == 4: the same input gives the same output, there's one right answer, and checking it is instant and free. Every one of those three comforts evaporates with LLMs:

Non-determinism. An LLM samples from a probability distribution over an effectively infinite space of possible responses. The same prompt yields different outputs across runs, temperatures, and model versions. There is no single "expected" value to assert against.
No ground truth. “Summarize this,” “answer this,” “write this” — open-ended tasks have many valid answers that differ in wording, length, and emphasis. Quality is a spectrum, not a pass/fail bit.
Subjectivity. Even humans don't agree on what's “good.” On genuinely subjective tasks, inter-annotator agreement is famously low — sarcasm detection routinely scores a Krippendorff's α below 0.35. If five experts disagree, whose grade is the ground truth?
Expensive to judge well. Generating an answer can be cheaper than judging one. Reliable evaluation often needs a stronger model (or a human) than the task itself — the opposite of how testing usually works.

Put together, the unit-test reflex doesn't just struggle — it's a category error:

# Same ticket, same prompt, run 3x at temperature 0.7 — three DIFFERENT,
# equally valid summaries (Claude, claude-opus-4-8):
for _ in range(3):
    print(summarize(ticket))
# -> "Double-charged in March; refund the duplicate. #4471"
# -> "User billed twice (Mar 3 & 12), one account, wants the dup refunded. Order 4471"
# -> "Refund request: duplicate March subscription charge, #4471"

# Now try the test you'd write for normal code:
assert summarize(ticket) == REFERENCE     # X fails — yet the output was GOOD
# There is no single REFERENCE. That one broken line is the whole problem:
# your unit-test intuition assumes one right answer, and here there isn't one.

That failing assert isn't a bug to fix. It's a sign you're holding the wrong mental model. You can't equality-check your way to quality when there's no single correct answer to check against.

See It: Grade It Yourself

Reading about “no single right answer” is one thing; feeling it is another. Below, you are the grader. Pick a task, read the model's output, and commit to a verdict — Pass, Borderline, or Fail. Then meet the five experts who graded the exact same output.

Interactive: pick an open-ended task (summarize a ticket / answer a question / judge a code explanation), grade the model's output yourself, then see how 5 expert graders scored the same output — they disagree — plus three other valid-but-different answers that exact-match grading would all reject.

Notice what happened: you formed a confident verdict, and reasonable experts landed somewhere else — and there were several other answers that were perfectly fine. That's not a defect in the graders. It's the nature of the task. “Good” is something you have to define, not something you can look up.

The Deeper Frame: The Three Gulfs

The sharpest map of why this is hard comes from Hamel Husain and Shreya Shankar (who train eval teams at OpenAI, Anthropic, Google, and Meta). They split the difficulty into three gulfs — three gaps every LLM application has to cross. Naming which gulf you're stuck in tells you which fix to reach for. Click through each:

Interactive gallery of the Three Gulfs of LLM development (Husain & Shankar): the Gulf of Comprehension (you vs your data), the Gulf of Specification (intent vs instructions), and the Gulf of Generalization (instructions vs the long tail) — each with how you cross it and a concrete example.

These aren't abstract. Most “the AI is broken” moments are really a comprehension failure (you never looked at the data), a specification failure (you never said what “good” means), or a generalization failure (it worked on your five examples and nothing else). Evals are how you measure your way across all three.

The Eval Itself Is Fragile: Prompt Sensitivity

Here's the twist that catches even careful teams. LLMs are wildly sensitive to tiny changes — and that includes the changes inside your eval. Swap a word (“Tell me” → “Please describe”), reorder your few-shot examples, change bullet points to prose, tweak the casing or a delimiter — and both the absolute scores and the model rankings can flip. There's a paper literally titled “A Single Character Can Make or Break Your LLM Evals.”

The lesson: a number from a single prompt template is not trustworthy on its own. Robust evaluation has to account for this fragility — testing across phrasings, averaging over variations, and never reading too much into one run of one template. The thing doing the measuring is made of the same unstable material as the thing being measured.

Public Benchmarks Won't Save You

“Can't I just check the model's MMLU score?” Public benchmarks are useful for model selection at a glance, but they cannot tell you whether your system works on your data — and they degrade in three predictable ways:

Goodhart's Law. When a measure becomes a target, it stops being a good measure. Once a benchmark is the thing everyone optimizes, models learn to ace the test without the underlying capability.
Saturation. Frontier models now cluster near the ceiling of the benchmarks that mattered two years ago. A test where everyone scores 96% can no longer tell good from great.
Contamination. If the test questions were in the training data, the score measures memorization, not reasoning. MMLU questions have been found verbatim in Common Crawl, the web dump most models pre-train on.

Treat public benchmarks as opinion polls, not thermometer readings: directionally useful, methodology-dependent, and trivially gameable. The eval that actually governs your product is the task-specific one you build from your own data — which is exactly where the next lessons go.

Why the Difficulty Is Good News

It would be easy to read all this as discouraging. It's the opposite. The difficulty is the moat.

Because LLM evaluation is genuinely hard, most teams don't do it — they ship on vibes, get stuck in reactive bug-whack, and stall. The teams that build the muscle to look at their data, specify what good means, and measure across the gulfs pull away and stay ahead. Hard does not mean impossible; it means valuable. Every obstacle in this lesson — non-determinism, subjectivity, the three gulfs, prompt sensitivity, gameable benchmarks — has a practical countermeasure, and the rest of this container is precisely that toolkit: metrics that fit open-ended tasks, LLM-as-judge done right, error analysis, and production observability.

You now understand the terrain. The point was never that it's too hard to measure — it's that measuring well is the skill that separates AI engineers from prompt tinkerers.

🧪 Try It Yourself

Run a tiny experiment that makes the subjectivity undeniable:

Take one open-ended output from something you've built — a summary, a support reply, a generated description.
Grade it yourself, in writing: Pass or Fail, and one sentence on why.
Ask a colleague to grade the same output — cold, no discussion, their own one-sentence reason.
Compare. Did you agree on the verdict? On the reason? Most pairs don't, fully.

That gap between your two reasonable judgments is the Gulf of Specification made visible — and it's exactly what a good rubric or eval has to pin down before it can grade anything reliably. If you couldn't write the criterion in one clean sentence, you just found why your evals (and your prompt) need work.

Mental-Model Corrections

“I'll just use exact match / BLEU / ==.” Only works when there's one right answer — which open-ended LLM tasks rarely have. Equality-checking rejects perfectly good outputs and lulls you into false confidence.
“A high benchmark score means it's good for my use case.” No — Goodhart, saturation, and contamination make public scores closer to opinion polls. Your task-specific eval on your data is the one that counts.
“One well-written eval prompt gives a reliable number.” No — prompt sensitivity means a single template can flip both scores and rankings. Test across variations; distrust any single run.
“Humans are the ground truth, so just ask people.” Partly — humans are essential, but they're inconsistent and disagree on subjective tasks, and they're slow and expensive. You need calibration and rubrics, not just “ask an annotator.”
“This is too hard — better to skip rigorous evals.” Exactly backwards. The difficulty is why it's a moat. Teams that measure across the gulfs win; teams that ship on vibes stall.

Key Takeaways

LLM evaluation breaks software-testing intuitions on purpose: outputs are non-deterministic over a near-infinite space, open-ended tasks have no single ground truth, “good” is subjective (experts disagree), and judging well can cost more than generating. output == expected is a category error.
The Three Gulfs name the difficulty: Comprehension (you can't see your own data), Specification (you can't precisely say what you want), and Generalization (it works on your examples, not the long tail). Diagnose which gulf you're in to pick the right fix.
The eval itself is fragile. Prompt sensitivity means a single word or format change can flip scores and rankings — never trust one template, one run.
Public benchmarks won't save you — Goodhart, saturation, and contamination (MMLU verbatim in Common Crawl) make them opinion polls, not thermometers. You need task-specific evals on your data.
The difficulty is the moat. Grading an LLM isn't reading an answer key — you design what “good” means. That design problem is hard, which is exactly why doing it well is the skill that wins.