Skip to main content
CodePeetCodePeet
LLM Evaluation & Observability

The Eval-Driven Development Mindset

Introduction

You can build an agent, wire up RAG, and craft a clever prompt. Now answer the question that decides whether any of it is real: how do you know it works?

Most AI features ship on vibes — someone tries a few inputs, the outputs look good, and it goes out the door. That's how demos are built. It is not how products are built. The single discipline that separates a flashy demo from a system you can actually trust, improve, and defend is evaluation — and the teams who do it well treat it not as a final QA step but as the engine of the whole engineering process.

This lesson is the mindset that the rest of this container is built on. It's a shift from “this looks good” to “this scores 0.91 and I can prove it.” Master this shift and you stop guessing; you start engineering.

An infographic titled 'The Eval-Driven Development Mindset' contrasting two ways to ship a change to an AI feature. On the left, labeled SHIP BY VIBES, a developer eyeballs three cherry-picked sample replies that all look good with green checkmarks and a thought bubble reading 'looks great, ship it', then deploys with a rocket — but a hidden regression reaches real users, shown by unhappy faces and the caption 'you only ever see what you sample'. On the right, labeled SHIP BY EVALS, the same change is run through a 50-case evaluation set that produces a single trustworthy score dropping from 0.85 to 0.71 with a red down-arrow, catching the regression BEFORE it ships, so the change is held; the caption reads 'the eval set sees all 50, every time'. Across the center is the definition: an eval equals an input plus a grader equals a score, described as the unit test of AI engineering. A horizontal band shows the eval-driven loop: Define success first, then Measure every change, then Improve, repeating — the eval is written BEFORE the prompt, the same way test-driven development writes the test before the code. Three cards explain: first, why vibes fail — large language models are non-deterministic, the same input can give different outputs, and no human can eyeball ten thousand traces, so the two or three you check are always the easy ones; second, evals do two jobs — capability evals ask 'can we do this at all' and climb from low pass rates, while regression evals ask 'can we still do this reliably' and must stay near one hundred percent to gate releases with no vibes-based approvals; third, evals are the moat — a team that trusts its eval numbers adopts new models in days instead of weeks, chases real gains instead of noise, and compounds a context-specific dataset competitors cannot copy. A takeaway banner reads: you cannot improve what you cannot measure; eval-driven development means defining success before you build, measuring every change, and letting the number, not the vibe, decide what ships.

The Vibes Trap

Picture the loop almost everyone starts with. You tweak a prompt, run two or three inputs, the replies read well, and you ship. It feels like progress. It is mostly luck, for two reasons that don't go away:

  • LLMs are non-deterministic. The same input can produce different outputs across runs, models, and temperatures. A unit test that checks output == expected doesn't survive contact with a system that has no single "expected." Eyeballing a couple of replies tells you almost nothing about the next thousand.
  • You can only ever sample the easy cases. Nobody can read 10,000 traces. So the 2-3 you check are the calm, typical ones — never the furious customer, the malformed input, the rare edge case. The failures hide exactly where you're not looking.

This is prompt-engineering-by-anecdote, and it's where AI projects quietly go off the rails: you "improve" the prompt, a few samples look better, and you ship a change that silently made things worse for the cases you never saw. The next lesson dissects why evaluating LLMs is so uniquely hard; here, just sit with the discomfort — you currently have no idea if your last change helped or hurt.

What an Eval Actually Is

An eval is refreshingly simple. In Anthropic's words, it's “a test for an AI system: give it an input, then apply grading logic to its output to measure success.” Three parts:

  1. Inputs — a set of representative cases (the more real, the better).
  2. A grader — logic that scores each output: did it pass? (a string check, a rule, a model-as-judge, a human — more on graders later in this container).
  3. A score — one number you can trust, track over time, and gate releases on.

Think of it as the unit test of AI engineering — adapted for a world where outputs vary. You're not asserting one exact string; you're measuring a pass rate over many cases. Here's an entire eval, start to finish, pointed at Claude:

import anthropic
client = anthropic.Anthropic()

# An eval is just three things: INPUTS + a GRADER + a SCORE.
# "Does the reply state the correct 30-day refund window, in <= 50 words?"
EVAL_SET = [                                    # 20-50 REAL cases, not 3 cherry-picked
    {"ticket": "How long do I have to return this?",        "must_include": "30 days"},
    {"ticket": "Can I get my money back after 3 weeks?",    "must_include": "30 days"},
    {"ticket": "I'm furious — this broke on day 25!",       "must_include": "30 days"},
    # ... +47 more, including the angry, weird, and edge cases you'd never eyeball
]

def run(ticket: str) -> str:                    # the system under test
    msg = client.messages.create(
        model="claude-opus-4-8", max_tokens=200,
        system="You are a support agent. Be accurate about the 30-day refund window.",
        messages=[{"role": "user", "content": ticket}])
    return msg.content[0].text

def grade(reply: str, must_include: str) -> bool:          # a DETERMINISTIC grader
    return must_include in reply and len(reply.split()) <= 50

score = sum(grade(run(c["ticket"]), c["must_include"]) for c in EVAL_SET) / len(EVAL_SET)
print(f"pass rate: {score:.0%}")     # -> ONE number you can trust, track, and gate on

That pass rate: 84% is the thing vibes can never give you: a trustworthy, repeatable number. Run it before a change and after, and for the first time you can prove whether you improved or regressed — across all 50 cases, not the 3 you happened to read.

Eval-Driven Development: Write the Eval First

Here's the mindset flip that gives this container its name. Don't bolt evals on at the end as a QA gate. Write the eval first — before the prompt, before the pipeline, before you even pick a model. It's Test-Driven Development for non-deterministic systems.

Why first? Because defining "good" as a testable spec forces a clarity you otherwise skip. As Anthropic puts it: “two engineers reading the same initial spec could come away with different interpretations — an eval suite resolves this ambiguity.” The moment you must turn "be helpful" into a grader, the vagueness becomes visible and you're forced to decide what helpful actually means.

# Eval-Driven Development: the eval comes FIRST — before the prompt, the
# pipeline, or the model. (Test-Driven Development, for non-deterministic systems.)

# 1. DEFINE "good" as a testable spec. Writing this forces product clarity —
#    two engineers with the same vague spec build two different things.
def passes(reply, case) -> bool:
    return case["fact"] in reply and is_polite(reply)

# 2. WRITE the eval set from real & expected cases (start with 20, not 1000):
EVAL = [{"ticket": "...", "fact": "30 days"}, ...]

# 3. ITERATE the prompt, MEASURING every change — the number says if it helped:
while pass_rate(EVAL) < 0.95:
    prompt = change_one_thing(prompt)
    print(pass_rate(EVAL))           # not "looks better" — 0.71 -> 0.83, or it didn't

# 4. LOCK it as a REGRESSION test: it must stay green on every future change,
#    every model swap, forever. That green bar is what lets you move fast.

Notice the shape: define → measure → improve, on repeat, with a number at every step. The eval isn't a phase that follows development — it shapes every iteration of it. The era of shipping AI features built on prompt-engineering-by-anecdote is ending; this is what replaces it.

See It: Vibes vs Evals

Time to feel the difference. Below, you've shipped a change to a support agent. You'll do exactly what a vibes-driven team does — eyeball three sampled replies and make the call yourself — and then watch the full eval set reveal what those three could never show. Try all three changes; one of them will surprise you.

Interactive: pick a change to a support agent, eyeball 3 sampled replies, then decide Ship or Hold. The full 50-case eval set then reveals the truth — a model swap that 'looks fine' actually drops the tone score 0.85 → 0.71, a regression on cases you never sampled. One change is a real win — but only the eval can prove it.

The lesson lands in your gut: vibes can only see what they sample, and you always sample the easy cases. Sometimes the vibe and the eval agree — but you can't know that without the eval. The number is the only thing that sees all 50.

Evals Have Two Jobs

As you adopt this mindset, you'll write two kinds of evals — and it helps to know which you're building:

  • Capability evals answer “Can we do this at all?” They start at a low pass rate and you push it up as you improve the system. (SWE-bench Verified, a coding-agent benchmark, climbed from ~40% to over 80% pass rate in a single year this way.)
  • Regression evals answer “Can we still do this reliably?” They sit near 100% and must stay there. Every prompt tweak, every model upgrade, every refactor runs against them — and a drop blocks the release.

That second category is the quiet superpower. With a regression suite, “eval thresholds decide whether a change ships — no vibes-based approvals.” It's the green bar that lets you move fast because you'll be caught the instant you break something.

Start Small — and Don't Wait

The most common reason teams don't have evals is a myth: that you need hundreds of perfectly designed cases first. You don't. Anthropic's guidance is blunt: start with 20-50 simple tasks drawn from real failures, not hundreds of perfect tests. Convert your manual spot-checks and bug reports directly into cases. Ten real examples beat zero perfect ones.

And start now, because “evals get harder to build the longer you wait.” Early on, product requirements translate naturally into test cases; later, you're reverse-engineering intent from a sprawling system. A tiny eval today compounds: every production failure you find becomes tomorrow's new case, and the set grows into something genuinely valuable — which is the whole point of the next section.

Why Evals Are the Real Moat

It's tempting to see evals as overhead. They're the opposite — they're often the most durable thing you build:

  • Velocity. Teams with a trusted eval suite “iterate weeks faster when new models arrive” — they point the suite at the new model and know in an hour whether to switch. Teams without one face prolonged, nervous testing cycles every time.
  • Signal over noise. “A team that trusts its eval numbers can focus on real improvements instead of chasing noise.” You stop arguing about whose anecdote is right.
  • A dataset competitors can't copy. Running evaluation at scale yields a “large, differentiated, context-specific dataset” — your real inputs, your failure taxonomy, your graders. That asset is your moat far more than any single prompt.

This is why "look at your data" is the gospel taught inside the frontier labs — Hamel Husain and Shreya Shankar, who train teams at OpenAI, Anthropic, Google, and Meta, reduce the entire field to it. And it's why AI Evals Engineer is now a distinct role. Evaluation isn't the unglamorous part of AI engineering. In 2026, it's the part.

🧪 Try It Yourself

Don't just read this — write one eval, right now, for something you've already built (a prompt, a RAG answer, an agent step). Five minutes:

  1. Grab 5 real inputs. Pull them from actual use, not your imagination — and make sure at least one is a hard/weird/angry case.
  2. Write the pass/fail criterion in one sentence. "A reply passes if it ___." Notice how hard this is — that struggle is the Gulf of Specification, and writing it down is the most valuable five minutes you'll spend.
  3. Predict your pass rate, then check. Guess the number first (say, "4 of 5"), then actually run the 5 and grade them. Were you right? The gap between your guess and the truth is exactly the gap vibes were hiding.

If writing the criterion made you realize you couldn't quite say what "good" means — congratulations, you just found the bug before your users did. That's eval-driven development.

Mental-Model Corrections

  • “Evals means benchmarks like MMLU.” No. Public benchmarks measure models; you need task-specific evals that measure your system on your data. Frontier models already saturate the old benchmarks — your eval set is the one that matters.
  • “I need a big, perfect eval set before it's worth it.” Backwards. 20-50 real cases beats waiting for the perfect 1000. The set is meant to grow; a 0% start is fine.
  • “Evals are a QA phase at the end.” No — they come first and shape every iteration (eval-driven development). Bolting them on last means you've already built the wrong thing.
  • “A good vibe check is basically the same thing.” It isn't, and it can't be — non-determinism plus the fact that you only sample the easy cases means vibes systematically miss regressions. The interactive above shows exactly how.
  • “Evals slow me down.” The opposite. A green regression bar is what lets you ship faster and more confidently, swap models in a day, and refactor without fear.

Key Takeaways

  • The core shift: from “this looks good” to “this scores 0.91 and I can prove it.” Eval-driven development is the discipline that turns AI demos into AI products.
  • An eval = inputs + a grader + a score — the unit test of AI engineering, adapted for non-deterministic systems. One trustworthy number beats a thousand anecdotes.
  • Vibes can't work, structurally: LLMs are non-deterministic and you can only ever sample the easy cases, so eyeballing systematically hides regressions.
  • Write the eval first (TDD for LLMs). Defining "good" as a testable spec forces the product clarity you'd otherwise skip — and the loop is define → measure → improve.
  • Two jobs: capability evals climb from low pass rates; regression evals stay near 100% and gate releases — no vibes-based approvals.
  • Start small (20-50 real cases), start now, and treat the set as a growing asset. A trusted eval suite is the moat: faster model adoption, signal over noise, and a dataset competitors can't copy. You can't improve what you can't measure.