Eval Anti-Patterns (Vibes-Based Testing)

Introduction

You have the mindset (The Eval-Driven Development Mindset), you respect the difficulty (Why Evaluating LLMs Is Uniquely Hard), you know the two modes (Offline vs Online Evaluation), and you can build a set (Building Your First Eval Set). Now the section's closer — the failure modes — because here's the uncomfortable truth: most teams who do evals still get fooled, just more confidently.

This is the rogues' gallery of eval anti-patterns: the recognizable smells, and the fix for each. The reason they're dangerous is uniform — every one of them produces a confident, wrong number, which is far worse than no number at all. Learn to spot these in your own work (and in code review), and you'll dodge the traps that quietly stall AI products everywhere.

An infographic titled 'Eval Anti-Patterns' presenting a rogues' gallery of the common ways teams fool themselves with evaluation, each paired with its fix. The headline anti-pattern is vibes-based testing: eyeballing a few outputs and shipping on it seems fine, which fails because of non-determinism and because you only ever sample the easy cases; the fix is a real eval set and a number. Goodhart's law is shown: when a measure becomes a target it stops being a good measure, producing models that please the evaluator while failing users; the fix is to remember the metric is a proxy, triangulate with multiple metrics, and keep reading real outputs. Overfitting is training on the test set: tuning the prompt against the same cases until they pass, so the eval climbs while production stays flat; the fix is a holdout you never optimize against, refreshed regularly. The tyranny of the single number is illustrated by an overall score of ninety-one percent that hides a refunds segment sitting at forty-one percent, because aggregation smooths away the long tail; the fix is to slice by segment. Three quieter traps are listed: set-and-forget, a stale dataset that never grows as the product and users shift, fixed by feeding it production failures; trusting the LLM judge blindly despite its position, verbosity, and self-preference biases, fixed by calibrating the judge against humans; and declaring victory on noise, shipping on a two-point move across twenty cases, fixed by using enough samples and repeated runs. Beneath all of them sits the cardinal sin: not looking at your data, trusting a number without ever reading the traces behind it; the one habit that prevents every other anti-pattern is to read your data on a regular cadence. The takeaway banner reads: every anti-pattern produces a confident, wrong number, the most dangerous kind, and the antidote that runs under all of them is to look at your data, slice it, and keep the set honest with reality.

#1 — Vibes-Based Testing (the mother of them all)

The most common anti-pattern needs little introduction by now: try a handful of inputs, eyeball the replies, decide it “looks good,” and ship. We dissected why it can't work in the mindset lesson — non-determinism means the same input varies, and you can only ever sample the easy cases, so eyeballing systematically hides regressions.

The fix is the whole point of this container: a real eval set and a trustworthy number. But here's the twist that motivates the rest of this lesson — graduating from vibes is not enough. Teams that build evals fall into subtler, more confident traps. The rest of the gallery is for them — for you, six months from now, sure you're doing it right.

#2 — Goodhart: the Number Goes Up, the Product Doesn't

Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. Point all your optimization pressure at a single metric and the model learns to win the metric without doing the job — the AI equivalent of teaching to the test.

In LLM-land this shows up as evaluator gaming: models that “please the evaluator while failing users.” If your LLM judge quietly rewards longer answers, your system learns to pad. If your metric rewards keyword overlap, it learns to stuff keywords. The score climbs; the product gets worse.

The fix: treat every metric as a proxy, never the goal. Triangulate — combine several metrics with regular qualitative review of real outputs, refresh the target so it can't be memorized, and stay suspicious of any number that's improving faster than your users' actual happiness.

#3 — Overfitting: Training on the Test Set

This one masquerades as diligence. You run the eval, tweak the prompt, run it again, tweak again — for two weeks — until the score is beautiful. The problem: you've been optimizing against your test cases, which is the eval equivalent of training on the test set. The number goes up because you've memorized the answers, not because the system got better. Production stays flat — or regresses — while your dashboard glows.

The fix: keep a holdout — a slice of cases you never iterate against, used only to check whether your gains are real. And refresh the set with new examples regularly, so you can't overfit a static target. If your eval score and your real-world results ever diverge, trust reality and suspect overfitting first.

#4 — The Tyranny of the Single Number

“We're at 91%.” It sounds like progress. It's often a smokescreen. A single aggregate is a smoothing function — it mathematically drowns out the signal of a segment that's failing badly. Your shiny 91% can hide a category sitting at 41%:

# The single-number trap, in code. Grade the set (with the Claude is_grounded()
# scorer from the last lesson), then SLICE - the average is a smoothing function
# that hides the segment that's on fire.
from collections import defaultdict

results = [(case, is_grounded(**case)) for case in EVAL_SET]   # graded by claude-opus-4-8

overall = sum(p for _, p in results) / len(results)
print(f"overall pass: {overall:.0%}")          # -> 91%   ("looks great - ship it!")

by_cat = defaultdict(list)
for case, passed in results:
    by_cat[case["category"]].append(passed)
for cat, ps in sorted(by_cat.items()):
    print(f"  {cat:10} {sum(ps)/len(ps):.0%}")
#   billing    96%
#   general    97%
#   refunds    41%    <- the regression the 91% average smothered

refunds 41% was invisible in the average — and refunds might be your highest-stakes flow. The fix is to slice: always break the number down by segment, category, intent, difficulty, user cohort. A single aggregate hides regressions inside one slice; per-segment breakdowns surface them. The headline number is for the standup; the sliced numbers are for the engineering.

See It: Diagnose the Eval

Recognizing anti-patterns in the abstract is easy; spotting them in a real setup is the skill. Below are three teams, each certain their evals are fine. For each, select every anti-pattern it commits — then check your diagnosis against the fixes.

Interactive: audit three realistic but flawed eval setups (a vibes-based team, a 0.95-accuracy team that overfit a stale single-number set, and a team that blindly trusts a biased LLM judge). Multi-select the anti-patterns each one commits, then check — caught (green), missed (red, with the fix), and wrongly-flagged (amber), with a score.

Notice how the traps cluster: a team that ships on vibes is usually also testing only the happy path and never reading data; a team obsessed with one number is usually also overfitting and letting the set go stale. Anti-patterns travel in packs — fix one and you'll often find its neighbors.

Three Quieter Traps: Stale Sets, Blind Judges, and Noise

A few more that are easy to miss precisely because everything looks rigorous:

Set-and-forget. You built a great eval set… eight months ago. Your product and your users have moved on; the distribution has shifted; the set now tests a world that no longer exists and silently stops catching regressions. Fix: treat the set as living — feed it production failures continuously, the offline↔online loop from the last lesson.
Trusting the judge blindly. LLM-as-judge is powerful, but judges have biases — they favor longer answers (verbosity), the first option shown (position), and their own style (self-preference) — and they report these flawed verdicts with high confidence. Trusting an uncalibrated judge means trusting a confident, biased number. Fix: calibrate the judge against human labels periodically; if agreement drops, tighten the rubric. (We'll do this properly in the LLM-as-a-Judge section.)
Declaring victory on noise. Your eval goes from 84% to 86% on 20 cases and you ship the “improvement.” But with non-determinism and a small set, a 2-point move is well within the noise band — you've mistaken randomness for progress. Fix: use enough samples, average over repeated runs, and ask whether the difference is actually significant before you believe it.

The Cardinal Sin: Not Looking at Your Data

Underneath every anti-pattern above is the same root cause: trusting a number without ever reading the traces behind it. Vibes skips the data. Goodhart happens because no one noticed the gamed outputs. The single number hides what a five-minute read would reveal. Stale sets persist because no one re-reads production. The judge's bias goes uncaught because no one compared it to reality.

So the one habit that prevents the entire gallery, the discipline the best practitioners reduce everything to: look at your data, on a regular cadence. Read real traces — the good, the bad, the weird — every week. (A close cousin of this sin is reaching for generic, off-the-shelf metrics instead of ones built from your observed failures; the cure is the same — let what you see in the data decide what you measure.) Numbers tell you that something changed; only your eyes on real outputs tell you what and why.

🧪 Try It Yourself

Audit a real eval — your own, or a teammate's — against this checklist. Be honest; the goal is to find the worst one and fix it this week:

Vibes? Is there an actual set and a number, or just “we tried some and it looked good”?
Single number? Can you produce a per-category breakdown right now — or only one aggregate?
Happy-path only? How many edge, adversarial, and negative cases are in the set?
Overfit? Is there a holdout you've never tuned against?
Stale? When did a new case last enter the set from production?
Judge uncalibrated? If you use an LLM judge, when did you last check it against a human?
Shipping on noise? Was your last “win” bigger than the run-to-run wobble?

Whichever question made you wince — that's your highest-leverage fix. Most teams flunk at least three. Naming them is how you stop being one of them.

Mental-Model Corrections

“My eval score went up, so the product got better.” Not necessarily — that's Goodhart / overfitting. A rising number can mean you taught to the test. Check against a holdout and real outputs.
“One headline accuracy number tells me how I'm doing.” It hides failing segments. Slice it — 91% overall can be 41% where it matters most.
“I have evals, so I'm safe from vibes.” Evals done wrong are confident vibes. Overfitting, stale sets, blind judges, and noise all produce trustworthy-looking, wrong numbers.
“The LLM judge is objective.” It has length, position, and self-preference biases and reports them confidently. Calibrate it against humans before you trust it.
“84% → 86% is an improvement.” On a small set, that's likely noise. Use enough samples and check significance before declaring victory.
“The number is the eval.” The number is a summary. The eval is incomplete until you've read the traces behind it — the cardinal habit that prevents every trap here.

Key Takeaways

Every anti-pattern produces a confident, wrong number — more dangerous than no number. The whole gallery is variations on fooling yourself.
Goodhart: when the metric becomes the target, models please the evaluator while failing users. Treat metrics as proxies; triangulate; keep reading outputs.
Overfitting: tuning against your test cases is training on the test set. Keep a holdout and refresh the set.
The single number lies by omission — aggregates smooth away failing segments. Slice by category, intent, cohort.
Quieter traps: stale set-and-forget (feed it production), blind LLM judges (calibrate vs humans), and declaring victory on noise (use enough samples).
The cardinal sin is not looking at your data. Every other anti-pattern is downstream of it. Read real traces on a regular cadence — it's the one habit that prevents them all, and the throughline of this entire section.