Agent Evaluation

Introduction

You've built a real agent: a brief and design (L6 — Project Brief & Design), tools and memory (L7 — Tools + Memory), and an MCP toolset you secured (L8 — MCP Server Integration). It runs. But here's the question that separates a demo from a product: is it actually any good — and how would you know?

Evaluating an agent is not the same as evaluating the RAG pipeline you graded back in L4 (Building the Evaluation Harness). There you scored a single answer against a reference. An agent doesn't produce one answer — it produces a trajectory: a sequence of decisions and tool calls on the way to the goal. A run can land on the right answer via a wrong, wasteful, or dangerous path — and answer-only eval will happily ship it. We call that a corrupt success, and catching it is the whole job.

In this lesson you'll build the evaluation that answers "is the agent good?" properly:

Grade on three levels — the final response (did it reach the goal?), the trajectory (did it take the right path?), and the single step (was each tool call the right tool with the right args?).
Use the right grader for each — deterministic trajectory match (cheap, fast, CI-friendly) where you know the expected path, and an LLM-as-judge (flexible) for open-ended runs.
Wire it into a gate — a golden trajectory dataset plus a CI check that blocks a regression, the same eval-driven loop you built in L4, now for agents.

Why this is a senior skill. Junior engineers eyeball a few runs and call it 'working'. Senior engineers build a repeatable, automated trajectory eval that catches loops, wrong-order calls, wasted steps, and silent tool-arg errors before users do. We use the course stack: the agentevals library for trajectory evaluators, Claude claude-sonnet-4-6 as the judge, and your existing LangSmith / Phoenix tracing.

Scope: this lesson owns evaluating the agent. Shipping it to production → L10 (Ship It).

Hero infographic titled 'Agent Evaluation' for the fourth lesson of the multi-tool agent project, on a white background. The deck says: an agent isn't one answer — it's a trajectory of tool calls; grade the path, not just the final text. The centre shows ONE agent run drawn as a chain with arrow connectors: user question -> search_docs -> web_search -> create_issue -> final answer. Three evaluation lenses point at that run, labelled L1 FINAL RESPONSE (did it reach the goal? — points at the answer box), L2 TRAJECTORY (did it take the right path? — wraps the whole chain), and L3 SINGLE STEP (right tool? right args? — points at one call). A small panel lists the agentevals trajectory_match_mode options: strict, unordered, subset, superset, plus tool_args exact vs ignore. A second grader is shown: an LLM judge (create_trajectory_llm_as_judge) that asks 'was this a logical, efficient, appropriate path?'. A warning badge reads: a right answer down a wrong path is a CORRUPT SUCCESS that outcome-only eval ships blind. Three summary cards along the bottom read: 'Deterministic trajectory match — cheap, fast, CI-friendly'; 'LLM-as-judge — flexible, for open-ended runs'; 'CI gate — block the PR when the trajectory score regresses'. A family strip lists the five project lessons — Brief & Design, Tools + Memory, MCP Server Integration, Agent Evaluation, Ship It — with the fourth highlighted. Slate, sky, cyan, violet, amber, emerald accents; generous legible text.

Why Evaluating an Agent Is Different

A RAG pipeline is a function: question in, answer out. You grade the output. An agent is a process: it reasons, picks a tool, reads the result, decides what to do next, maybe loops, and eventually answers. Two runs that reach the same answer can have wildly different paths — and the path is where agents fail.

Consider one task — "look up our SLA, check the live status page, then file a ticket about any discrepancy" — and four runs that all return a plausible answer:

Clean: search_docs -> web_search -> create_issue. Right tools, right order, right args. ✓
The looper: calls search_docs and web_search twice each before answering. Correct outcome — but slow, expensive, and fragile. A corrupt success.
Wrong order: files the ticket before reading the docs or status. It happened to work; on a different input it files a garbage ticket.
Wrong arg: right tools, right order — but create_issue ran with the wrong title/body. The answer text looks fine; the side effect is wrong.

Outcome-only evaluation passes all four. Only by grading the trajectory do the last three show up. This is the core mental shift: for an agent, the path is part of the output. A right answer reached by a wasteful, mis-ordered, or off-policy path is a latent bug — and because agents take real actions (write tickets, send emails, spend money), a bad path isn't just inefficient, it can be harmful. That's why we evaluate the journey, not only the destination.

The Three Levels of Agent Evaluation

Don't pick one kind of eval — use three levels together, because each answers a different question and catches a different class of bug:

Level	Question it answers	What it catches	How you grade it
1 · Final response	Did it reach the goal?	wrong/missing answer, task failure	reference answer match, or an LLM judge (your L4 metrics)
2 · Trajectory	Did it take the right path?	loops, wrong order, wasted steps, missing/extra tools	trajectory match vs a gold path, or an LLM judge over the run
3 · Single step	Was each call correct?	wrong tool selection, wrong parameters	per-call tool-selection + arg-accuracy checks

The intuition: final response tells you what went wrong; trajectory tells you where; single step tells you which decision. A failing final response with a passing trajectory points you at the task or the tools; a passing final response with a failing trajectory is a corrupt success you'd otherwise ship.

Alongside the three correctness levels, track system efficiency — latency, tokens, number of tool calls, and convergence (how close the run's length is to the optimal path). Quality first; then optimize cost and speed without regressing quality. The interactive later in this lesson shows all three levels plus efficiency updating live as you mutate a run.

Building a Golden Trajectory Dataset

Evaluation needs a fixed, versioned set of tasks — the agent equivalent of the golden set you curated in L4 (Building the Evaluation Harness). The difference: each example carries not just an expected answer but an expected trajectory (the reference path of tool calls). Aim for 20–50 tasks that span your agent's real jobs — and deliberately include the hard, multi-tool, must-use-memory, and ambiguous cases, because that's where trajectories break.

A trajectory is just a list of messages — OpenAI-style dicts (or LangChain messages; agentevals accepts both). Each example pins the path you'd accept:

# eval/agent_golden.py
GOLDEN = [
    {
        "question": "What's our SLA, and does the live status page agree?",
        "reference_answer": "Our SLA is 99.9% uptime; the status page currently agrees.",
        # the path you'd accept: read internal docs, then external status, then (only if needed) act
        "trajectory": [
            {"role": "user", "content": "What's our SLA, and does the status page agree?"},
            {"role": "assistant", "content": "", "tool_calls": [
                {"function": {"name": "search_docs", "arguments": '{"query": "SLA uptime"}'}}]},
            {"role": "tool", "content": "SLA: 99.9% uptime."},
            {"role": "assistant", "content": "", "tool_calls": [
                {"function": {"name": "web_search", "arguments": '{"query": "status page"}'}}]},
            {"role": "tool", "content": "All systems operational."},
            {"role": "assistant", "content": "SLA is 99.9% and the status page agrees."},
        ],
    },
    # ... 20-50 tasks: multi-tool, memory-dependent, ambiguous, and adversarial cases
]

Where do reference trajectories come from? Three sources, in order of value: (1) hand-authored for the critical tasks (you decide the right path); (2) captured from a known-good run of your agent and then reviewed (trace -> dataset, exactly what LangSmith/Phoenix make easy); (3) synthetic variations for breadth. Curate deliberately — a golden set the agent can't fail teaches you nothing. Store it next to your code and version it: when you change the agent, the dataset is the contract it must still satisfy.

Deterministic Trajectory Match with agentevals

The fastest, cheapest evaluator compares the agent's actual trajectory to your reference without an LLM — pure structural matching. agentevals gives it to you ready-made. The key decision is the trajectory_match_mode, because how strict to be depends on how many valid paths exist:

strict — same tools, same order, same args. Use for fixed pipelines where the path is the spec.
unordered — same set of tool calls, any order. Use when steps are independent (e.g. two lookups in either order).
superset — the agent called at least the reference tools (extra allowed). Use for "it must do these, more is OK."
subset — the agent called only tools within the reference (no off-script tools), though it may do fewer. Use to assert "never call anything outside this set."

And tool_args_match_mode (exact vs ignore) decides whether the arguments must match too — exact catches the wrong-arg bug; ignore grades the shape of the path only.

# eval/trajectory_match.py
from agentevals.trajectory.match import create_trajectory_match_evaluator

# pick the mode that matches how much path-flexibility is actually valid for the task
strict    = create_trajectory_match_evaluator(trajectory_match_mode="strict")
unordered = create_trajectory_match_evaluator(trajectory_match_mode="unordered")
superset  = create_trajectory_match_evaluator(
    trajectory_match_mode="superset",
    tool_args_match_mode="ignore",   # grade tool *selection*, not exact arguments
)

# run YOUR agent (from L6-L8), then grade its messages against the reference path
result = agent.invoke({"messages": [("user", task["question"])]})
ev = strict(outputs=result["messages"], reference_outputs=task["trajectory"])
print(ev)
# -> {'key': 'trajectory_strict_match', 'score': True, 'comment': None}

Two things to notice. One: you pass result["messages"] directly — agentevals accepts a list of LangChain messages or OpenAI-format dicts and normalizes them for you. Two: the result is a boolean score with a stable key — deterministic, instant, and free of LLM cost or variance, which makes it perfect for a CI gate. Reach for trajectory match whenever you know the path you expect; reserve the (slower, pricier) LLM judge for when you don't.

LLM-as-Judge Over the Trajectory

Many good agents have many valid paths — no single reference captures them. For those, grade the run qualitatively with an LLM judge that reads the whole trajectory and asks: was this a logical, efficient, appropriate way to reach the goal? agentevals ships this too, and you point it at your course judge model, Claude claude-sonnet-4-6 (the model string goes through LangChain's init_chat_model):

# eval/trajectory_judge.py
from agentevals.trajectory.llm import (
    create_trajectory_llm_as_judge,
    TRAJECTORY_ACCURACY_PROMPT,                 # reference-free: judge the path on its own merits
    TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,  # or compare against a gold path
)

judge = create_trajectory_llm_as_judge(
    prompt=TRAJECTORY_ACCURACY_PROMPT,
    model="anthropic:claude-sonnet-4-6",       # your course model as the judge
)

result = agent.invoke({"messages": [("user", task["question"])]})
verdict = judge(outputs=result["messages"])    # no reference needed
print(verdict)
# -> {'key': 'trajectory_accuracy', 'score': True,
#     'comment': 'The agent searched internal docs, then verified against the live status
#                 page, and answered without redundant calls. Logical and efficient.'}

The judge returns a score and a comment explaining its reasoning — that explanation is gold for debugging why a path was marked weak. Use TRAJECTORY_ACCURACY_PROMPT when there's no single right path (judge the run on its own merits) and ..._WITH_REFERENCE when you have a gold trajectory to compare against. You can pass continuous=True for a 0–1 score instead of pass/fail, and add few_shot_examples=[...] to align the judge with your taste — the same calibration discipline you applied to LLM judges in L4. Rule of thumb: deterministic match where the path is known; LLM judge where it's open-ended — and often both, as complementary signals.

Single Step, Tool-Calling Accuracy & Efficiency

Trajectory match grades the path as a whole; single-step evaluation zooms into one decision at a time for diagnostic precision. Two questions per call:

Tool selection — given the state at that step, was the right tool chosen? (Picking web_search when the answer is in internal docs is a selection error even if the run still works.)
Parameter accuracy — were the arguments extracted correctly from the request? (create_issue with the wrong title, refund with $300 instead of $30.)

Because every tool call is captured as a span by your OpenInference/Phoenix tracing (set up in the observability lessons), you can run an eval over those spans — Phoenix ships a tool-calling eval template that checks exactly the "did it extract the right parameters?" question against each llm.function_call. You can also write a tiny deterministic check for the cases you care about:

# eval/single_step.py — grade tool SELECTION + PARAM accuracy for one step
def grade_step(call: dict, expected: dict) -> dict:
    right_tool = call["name"] == expected["name"]
    right_args = right_tool and call["args"] == expected["args"]
    return {"tool_selection": right_tool, "param_accuracy": right_args}

# convergence / efficiency: how close was the run to the optimal path?
def convergence(actual_steps: int, optimal_steps: int) -> float:
    return min(optimal_steps / max(actual_steps, 1), 1.0)   # 1.0 = no wasted steps

Then track system efficiency next to correctness: latency, tokens, tool-call count, and convergence (optimal / actual path length — Phoenix calls this a convergence eval, comparing the ideal path length to the run's). A looping agent scores 1.0 on outcome but low on convergence — precisely the corrupt-success signal you want surfaced. Optimize efficiency only once quality holds; a cheaper agent that gets more answers wrong is not cheaper.

Running the Suite — LangSmith, Phoenix & a CI Gate

Now assemble the levels into one suite that runs over the whole golden set and reports a scorecard. LangSmith's evaluate() runs your agent over a dataset and applies each evaluator; your agentevals evaluators drop straight in. Wrap each so LangSmith feeds it the run's messages and the example's reference trajectory:

# eval/run_suite.py
from langsmith import Client
from agentevals.trajectory.match import create_trajectory_match_evaluator
from agentevals.trajectory.llm import create_trajectory_llm_as_judge

client = Client()

def run_agent(inputs: dict) -> dict:
    msgs = agent.invoke({"messages": [("user", inputs["question"])]})["messages"]
    return {"messages": msgs}

_match = create_trajectory_match_evaluator(trajectory_match_mode="superset")
_judge = create_trajectory_llm_as_judge(model="anthropic:claude-sonnet-4-6")

def trajectory_match(run, example):       # L2 — deterministic
    return _match(outputs=run.outputs["messages"],
                  reference_outputs=example.outputs["trajectory"])

def trajectory_quality(run, example):     # L2 — LLM judge
    return _judge(outputs=run.outputs["messages"])

client.evaluate(
    run_agent, data="agent-golden",
    evaluators=[trajectory_match, trajectory_quality],   # + final-response + single-step evaluators
)

Every run is also traced — LangSmith (or Phoenix via OpenInference) captures each tool call as a span, so a failing score links straight to the exact step that broke. That trace-to-eval loop is what makes debugging an agent tractable. Finally, turn the suite into a gate: a CI job that runs the eval on every PR and fails the build if a metric regresses — the same eval-driven discipline from L4, now guarding trajectories instead of answers.

Wiring It All Together

Pull the levels into one module and a CI gate. The gate runs the suite, averages each metric over the golden set, and asserts against targets — so a regression in trajectory quality blocks the PR just like a regression in answer quality did in L4:

# eval/test_agent_gate.py   ->  runs in CI (pytest); blocks the PR on regression
import statistics

from eval.agent_golden import GOLDEN
from eval.suite import score_agent          # runs all 3 levels for one task

TARGETS = {
    "final_response":      0.90,   # L1 — reached the goal
    "trajectory_superset": 0.90,   # L2 — called the right tools
    "trajectory_accuracy": 0.85,   # L2 — judge: logical/efficient/appropriate
    "param_accuracy":      0.95,   # L3 — right args
    "convergence":         0.80,   # efficiency — few wasted steps
}

def test_agent_meets_targets():
    scores = {m: [] for m in TARGETS}
    for task in GOLDEN:
        for metric, value in score_agent(task).items():
            scores[metric].append(value)
    for metric, target in TARGETS.items():
        mean = statistics.mean(scores[metric])
        assert mean >= target, f"{metric} {mean:.2f} < target {target} — regression, blocking merge"

That's the whole loop: a golden trajectory dataset, three levels of evaluators (deterministic match + LLM judge + single-step), efficiency tracking, full traces, and a CI gate that holds the line. Your agent is no longer 'working because it looked fine in three runs' — it's measured, and every change is measured against the same bar. Next, L10 (Ship It) takes this measured agent to production.

✅ Definition of Done (this step)

Before L10, your agent should be measured, not vibe-checked:

A golden trajectory dataset (20–50 tasks) with a reference answer and trajectory each, including hard/multi-tool/memory/ambiguous cases, versioned with your code.
You can name and apply the three levels — final response, trajectory, single step — and say what each catches.
Deterministic trajectory match wired with agentevals, and you can justify your trajectory_match_mode (strict / unordered / subset / superset) and tool_args choice per task.
An LLM-as-judge (create_trajectory_llm_as_judge, model claude-sonnet-4-6) for open-ended paths, with at least a couple of few_shot_examples to calibrate it.
Single-step checks (tool selection + param accuracy) and efficiency metrics (latency, tokens, tool-calls, convergence) reported.
The suite runs over the dataset (LangSmith evaluate / Phoenix), with traces, and a CI gate that fails the build on a regression.

If you can point at a number for "how good is the agent's path, not just its answer?" — and a CI check that defends it — you're ready to ship in L10.

See It — The Agent Eval Studio

This studio puts all three levels in your hands at once. Pick a task (each has a gold reference trajectory), then build the agent's actual run: click tools from the palette to append calls, reorder them (▲▼), corrupt a call's argument, or remove a step — or load a preset.

Start on ideal — every level PASS, the judge gives 1.00, verdict Ship.
Load wrong order — note strict ✗ but unordered ✓ (same tools, bad sequence), and the judge flags appropriate ✗ (it acted before it read). Final response still passes → a corrupt success.
Load wrong arg, then flip tool_args to ignore — watch the trajectory match flip to PASS while the final response stays FAIL. That's the trap of ignoring args.
Load wrong tool (delete_repo) — caught at every level; the judge marks it illogical (off-policy).

The Agent Eval Studio — grade an agent run the way agentevals does, live, on three levels at once. Pick a task (each has a gold reference trajectory), then BUILD the agent's actual run: click tools from the palette to append calls, reorder them, corrupt an argument, or remove a step — or load a preset (ideal / wasteful loop / wrong order / wrong tool / wrong arg). Watch all three levels update: L1 final response (did it reach the goal?), L2 trajectory match (flip trajectory_match_mode between strict / unordered / subset / superset and tool_args between exact / ignore), and L3 single step (right tool? right args? per call). An LLM-judge panel scores the path on logical / efficient / appropriate, and an efficiency readout shows optimal vs actual steps, wasted steps, and the convergence ratio. The lesson lands in your hands: a wrong-order run fails strict but passes unordered; ignoring args lets a wrong-argument run pass the trajectory match while the final response still fails; an off-policy delete_repo call is caught at every level. A right answer down a wrong path is a corrupt success. Deterministic teaching model, not a live agent.

Notice three things. One: the mode is a judgment call — strict for fixed pipelines, unordered/superset when many paths are valid; the same run passes or fails depending on what you should accept. Two: levels disagree on purpose — a passing final response with a failing trajectory is exactly the corrupt success you're hunting. Three: deterministic and judge are complementary — the match is cheap and exact, the judge explains why a path was weak.

🧪 Try It Yourself

Reason these out, then check against the studio and the code.

1. Your agent answers a question correctly but called the search tool four times when once would do. Final-response eval says PASS. Which level catches the problem, and which efficiency metric quantifies it?

2. A task has two independent lookups that can happen in either order. Your agent does them in the opposite order from your reference. Which trajectory_match_mode should you use so this doesn't fail — and which mode would (wrongly) fail it?

3. You set tool_args_match_mode="ignore" and your wrong-argument run starts passing the trajectory match. Is the agent fixed? What does this tell you about that setting?

4. When should you use deterministic trajectory match versus an LLM-as-judge — and why might you run both on the same run?

5. Why is a golden trajectory dataset + CI gate better than carefully reading 10 agent traces by hand before each release?

Answers.

1. Level 2 (trajectory) — specifically the efficiency / convergence metric: optimal / actual path length drops well below 1.0 (and tool-call count and tokens spike). The answer is right, so it's a corrupt success; only path-level eval surfaces the waste.

2. Use unordered (same set of tools, any order) — or superset. strict would fail it, because strict requires the exact same order. Match the mode to how much path-flexibility is genuinely valid for the task; over-strict evals generate false alarms and train you to ignore them.

3. No — the agent is not fixed. ignore tells the evaluator to skip argument comparison, so it only grades the shape of the path; the wrong argument is still wrong. You'd see this immediately because the final-response and single-step (param accuracy) levels still FAIL. Lesson: tool_args="ignore" grades tool selection, not correctness — don't mistake a passing match for a correct run.

4. Deterministic match when you know the expected path — it's instant, free, reproducible, and perfect for a CI gate. LLM-as-judge when the path is open-ended (many valid routes) or you want a qualitative read on efficiency/appropriateness — it's flexible but slower, costs money, and varies. Run both because they're complementary signals: the match gives a hard pass/fail on structure, the judge gives a graded, explained verdict on quality.

5. A dataset + gate is repeatable, automated, and objective — it runs the same tasks on every change, scores them the same way, and blocks regressions before merge. Hand-reading traces is slow, inconsistent, doesn't scale, and silently drifts; you'll miss the one trajectory that broke. Eval is infrastructure, not a pre-release ritual — the same lesson as L4, now for agents.

Mental-Model Corrections

"If the final answer is right, the agent is good." → Not for an agent. A right answer via a wasteful, mis-ordered, or off-policy path is a corrupt success — outcome-only eval ships it blind. Grade the trajectory.
"Agent eval is just RAG eval again." → RAG eval grades one answer; agent eval grades a trajectory of decisions and tool calls. New failure modes: loops, wrong order, wrong tool, wrong args, wasted steps.
"Use the LLM judge for everything." → Use deterministic trajectory match where you know the path — it's cheaper, faster, reproducible, and CI-friendly. Reserve the judge for open-ended runs.
"strict is the safest mode." → strict is the most brittle. If many paths are valid, strict throws false failures you'll learn to ignore. Pick the mode by how much flexibility is genuinely valid: unordered/superset when several routes are fine.
"tool_args=\"ignore\" makes the eval pass, so the run is fine." → ignore skips argument checking; the wrong arg is still wrong (final-response + single-step still fail). It grades tool selection, not correctness.
"One number tells me if the agent is good." → Use three levels (final / trajectory / single step) plus efficiency. They catch different bugs and sometimes disagree on purpose — the disagreement is the signal.
"I'll just eyeball a few runs before shipping." → Manual spot-checks don't scale, drift, and miss the broken trajectory. Build a golden dataset + CI gate — eval is infrastructure.
"Efficiency means make it cheaper." → Efficiency is secondary to quality. A cheaper agent that gets more answers wrong isn't cheaper. Optimize tokens/latency/steps only once quality holds.

Key Takeaways

An agent is a trajectory, not an answer. Grade the path — a right answer down a wrong path is a corrupt success that outcome-only eval ships blind.
Evaluate on three levels: final response (reached the goal?), trajectory (right path?), and single step (right tool, right args?) — each catches a different class of bug; they sometimes disagree on purpose.
Two graders, by job. Deterministic trajectory match (create_trajectory_match_evaluator) is cheap, fast, reproducible, CI-friendly — use it where you know the path; pick the mode (strict / unordered / subset / superset) and tool_args (exact / ignore) to match valid flexibility. LLM-as-judge (create_trajectory_llm_as_judge, model claude-sonnet-4-6) is flexible — use it for open-ended paths. Often run both.
Track efficiency next to correctness — latency, tokens, tool-call count, and convergence (optimal/actual path length). Optimize only once quality holds.
Make eval infrastructure. A versioned golden trajectory dataset + traces (LangSmith / Phoenix) + a CI gate that blocks regressions — the L4 discipline, now for agents.
Next — L10 (Ship It): take this measured agent to production — packaging, deployment, monitoring, and the online evaluation that watches its trajectories in the wild.