Evaluating Agents: Trajectories, Tool Correctness & Step Efficiency

Introduction

You can now see what an agent did — every span of its trace (L177 (Tracing LLM & Agent Calls)). This lesson is about judging it. And agents need a fundamentally different kind of evaluation than a single LLM call, because an agent doesn't produce one output — it takes a multi-step path of reasoning and tool calls to reach a goal.

The trap is to evaluate only the final answer. An agent can call every tool, return a perfectly correct response, and still have done something fragile, wasteful, or dangerous along the way — it looped three times, reserved inventory before checking stock, or refunded $300 instead of $30 and got lucky that nobody noticed. Conversely, it can take a flawless path and still miss the goal. So you judge an agent on two axes: the outcome (did it achieve the goal?) and the trajectory (was the path sound?). This lesson builds the metrics for both — trajectory match, tool correctness, and step efficiency — read straight from the traces you learned to capture.

An infographic titled 'Evaluating Agents: Trajectories, Tool Correctness, and Step Efficiency' that explains why an agent must be judged on two axes, not one. Because an agent takes a multi-step path of tool calls, judging only the final outcome is not enough: you also judge the process, the trajectory it took. The two axes are outcome, meaning did the agent achieve the user's goal, and process, meaning was the path sound with the right tools, right arguments, sensible order, and no wasted steps. Crossing them gives four quadrants. Correct outcome with a good process means ship it. Correct outcome with a bad process is a corrupt success, a right answer reached through a wasteful, looping, or risky path that is fragile and expensive and that outcome-only evaluation would have shipped blind. A failed outcome with a good process means the gap is the task or the tools, not the trajectory. A failed outcome with a bad process is simply broken. To score the trajectory against a gold reference there are three match modes: strict requires the same tools in the same order with the same arguments; unordered requires the same tool calls in any order, useful when many valid paths exist; and superset requires the agent to have called at least the gold tools, allowing extras. You pick the mode by how much path flexibility is genuinely valid, because agents can succeed through multiple valid paths so an exact match is often too strict. Tool correctness has two parts: tool selection, meaning did it call the right tool, and argument correctness, meaning did it pass the right parameters, because a right tool called with a wrong argument, like refunding three hundred dollars instead of thirty, is a real failure that a name-only check misses. These are summarized as tool-call precision, the fraction of called tools that were correct, and recall, the fraction of needed tools that were covered. Step efficiency penalizes unnecessary steps, retries, and loops, since a looping agent wastes cost and latency and is fragile. When there is no single gold path you judge the trajectory with an LLM judge on reasoning quality, plan adherence, and efficiency. Common agent failure modes read from the trace include wrong tool selection, wrong arguments, looping, plan drift, premature stopping, and error compounding. The takeaway banner reads: evaluate an agent on outcome and trajectory together; pick a match mode by how much path flexibility is valid; check tool selection and arguments; penalize wasted steps; a right answer down a bad path is a corrupt success.

Outcome Is Not Enough: The Two Axes

Outcome evaluation asks the blunt question: did the agent's full run accomplish the user's goal? It's usually binary — pass/fail on the final state — or an LLM-judge that infers the task, reads the steps and final answer, and decides if the goal was met. It's essential, but it's blunt: it tells you something went wrong, not where or why.

Trajectory (process) evaluation asks the richer question: was the path sound? It scores the intermediate steps — which tools were chosen, with what arguments, in what order, and how efficiently. “This is where the real signal lives.” Cross the two axes and you get four cases:

	Process good	Process bad
Outcome ✓	✓ Ship	⚠ Corrupt success
Outcome ✗	task / tools gap	✗ Broken

The dangerous cell is the top-right: corrupt success — a right answer reached through a bad path. Outcome-only eval marks it green and you ship an agent that's secretly fragile, slow, and expensive. “An agent can call every tool correctly and still fail” — and it can also succeed while doing something you'd never approve of. You need both axes.

Trajectory Match: strict, unordered, superset

The most direct way to score a trajectory is to compare it to a gold reference — the tool-call sequence an expert would take. But agents have a wrinkle single LLM calls don't: there are often many valid paths. For a coding task the agent might search the docs first, then write a skeleton, then test — or write the skeleton and test as it goes. Both are correct. So you choose how strict the comparison is:

strict — same tools, in the same order, with the same arguments. Tightest. Use it when order matters (you must check_inventory before reserve_item).
unordered — the same set of tool calls, in any order. Use it when the agent legitimately has freedom in how it gathers information, but you still care that all the needed calls happened.
superset — the agent called at least the gold tools (extras allowed). The most lenient — it confirms nothing required was missed, but it won't catch wasted steps (that's what efficiency is for).

The rule: pick the mode by how much path-flexibility is genuinely valid. strict is often too brittle for agents (it fails any valid reordering); unordered/superset (or an LLM judge) fit the multi-path reality better. Here's the evaluator:

# Agents are judged on the PATH, not just the answer. Compare the actual tool-call
# trajectory to a GOLD reference. Pick the mode by how much path-flexibility is valid.
from agentevals.trajectory.match import create_trajectory_match_evaluator

evaluator = create_trajectory_match_evaluator(
    trajectory_match_mode="strict",     # "strict":    same tools, same ORDER, same args
                                        # "unordered": same tool calls, ANY order
                                        # "superset":  agent called AT LEAST the gold tools
    tool_args_match_mode="exact",       # compare ARGS too -> a right tool + wrong arg = FAIL
)
result = evaluator(outputs=agent_trajectory, reference_outputs=gold_trajectory)
# result["score"] -> True / False for this mode

# Tool correctness as PRECISION / RECALL over the tool calls:
def tool_pr(actual, gold):
    correct = [c for c in actual if any(c.tool == g.tool and c.args == g.args for g in gold)]
    precision = len(correct) / len(actual)                       # of what it called, % right
    recall    = len({c.tool for c in correct}) / len({g.tool for g in gold})  # of needed, % covered
    return precision, recall   # low precision = wasted/wrong calls; low recall = missed a step

One subtlety the modes share: by default, two tool calls are “equal” only if they have the same arguments to the same tool. Which brings us to the half of tool correctness everyone forgets.

Tool Correctness: Selection and Arguments

Tool correctness has two parts, and skipping the second is the classic agent-eval bug:

Tool selection — did the agent call the right tool? (Compare the tools it called to the ideal set for the input.)
Argument correctness — did it pass the right parameters? A right tool with a wrong argument — issue_refund(amount=300) when the user asked for $30 — is a real, shippable failure that a name-only check waves straight through.

You summarize selection as precision and recall over the tool calls:

Tool precision = of the tools the agent called, what fraction were correct? (Low precision → wasted or wrong calls.)
Tool recall = of the tools it should have called, what fraction did it cover? (Low recall → it skipped a needed step.)

And you score arguments separately — often as partial credit (the percentage of correct parameters), not just exact match. The practical guidance: always compare arguments, allow order independence and repeat calls where they're functionally fine, but flag a wrong-argument call as a hard failure. In the widget you'll watch a wrong-\$300-refund run pass every name-only mode and fail the moment you turn argument comparison on.

See It: Grade the Agent

Grade four agent runs yourself. Compare each agent's actual tool-call trajectory (a chain with arrow connectors) to the gold reference, then flip the match mode (strict / unordered / superset) and toggle compare args — and watch the metrics and the outcome × process verdict change. Find the corrupt success, the order bug that only strict catches, and the wrong-argument refund that only argument-comparison catches.

Interactive: an agent-trajectory grader. Four agent runs as tabs, each showing the GOLD reference trajectory and the agent's ACTUAL trajectory as chains of tool-call nodes connected by arrows (tool name + arguments per node), with mismatched steps colored (ok green / ↻ duplicate amber / ✗ wrong-arg red / out-of-order flagged). The user flips two controls: the trajectory MATCH MODE (strict / unordered / superset) and COMPARE ARGS (on/off). Live metrics update: outcome (PASS/FAIL), the selected-mode match (PASS/FAIL), tool precision, tool recall, arg accuracy, and step efficiency; a per-mode strip shows strict/unordered/superset ✓/✗ simultaneously; and a 2x2 outcome×process verdict renders (Ship / Corrupt success / task-tools gap / Broken). Run 1 'the looper': right answer but search_orders and get_status are each called twice (loop) → outcome PASS but efficiency 60%, strict ✗ and unordered ✗ but superset ✓ → a CORRUPT SUCCESS (outcome-only eval would ship it blind). Run 2 'wrong order': reserved inventory before checking it → same tools so unordered ✓ and superset ✓, but strict ✗ because order matters here (risk of overselling). Run 3 'wrong arg': refunded $300 instead of $30 → with compare-args ON every mode FAILS and outcome FAILS; toggle compare-args OFF and the modes falsely PASS, teaching that you must compare arguments. Run 4 'clean': right tools, right args, right order, no waste → all modes ✓, outcome ✓ → Ship. Teaches: evaluate outcome AND trajectory; pick the match mode by valid path-flexibility; tool correctness = selection + args (precision/recall); penalize wasted steps; a right answer down a bad path is a corrupt success.

Two toggles, four runs, and the whole lesson is in your hands: the looper that ships green under outcome-only eval, the reorder that only strict flags, and the $300 refund that only argument comparison catches. The match mode and the args switch aren't cosmetic — they decide which real failures your agent eval can even see.

Step Efficiency: The Cost of a Bad Path

A correct answer down a wasteful path is still a problem — in production it's money, latency, and fragility. Step efficiency measures whether the agent reached the goal “through the most direct path possible, without performing inefficient actions.” You score it two ways:

Deterministically — the fraction of redundant tool calls (repeats, retries, loops) relative to the total, or len(gold_steps) / len(actual_steps). The looper in the widget scores 60% efficiency: it did 5 calls where 3 would do.
With an LLM judge — extract the user's goal, look at the trajectory and the available tools, and rate how much of the path was actually necessary (“the amount of unnecessary evidence collected”).

Why it matters beyond the bill: every extra step is another chance to drift off plan, compound an error, or trip a rate limit. An agent that takes 12 tool calls to do a 3-call job isn't thorough — it's unreliable. Efficiency is the metric that turns “it worked” into “it worked, repeatably, at a sane cost.” This is also why the superset match mode isn't enough on its own — it permits extras, so you pair it with an efficiency check.

When There's No Gold Path: LLM-as-Judge

For open-ended agents, writing a single gold trajectory for every input is impossible — there are too many valid paths. The answer is the reference-free evaluator from L166–L169: an LLM-as-judge that reads the whole trajectory and scores it directly on rubric dimensions — reasoning relevancy (does the reasoning behind each tool call tie to the user's request?), reasoning coherence (does it progress logically?), plan adherence, and efficiency.

Frameworks like AdaRubric take this per-step: score each trajectory step on correctness, efficiency, safety, and reasoning, each with a confidence weight, so a single bad step doesn't get averaged away. And task completion itself is often an LLM judge — infer the goal from the input, analyze the steps and final response, and decide if the goal was achieved (this scales to open-ended tasks with no ground-truth dataset).

# No single gold path? JUDGE the trajectory with an LLM (reference-free), score efficiency,
# and ALSO check the outcome. Gate on BOTH.
from agentevals.trajectory.llm import create_trajectory_llm_as_judge

judge = create_trajectory_llm_as_judge(model="claude-opus-4-8",
    prompt="Did the agent use the right tools, in a sensible order, with no wasted steps, and "
           "reasoning that ties to the user's goal? Reply pass/fail with a reason.")
process_ok = judge(outputs=agent_trajectory)        # PROCESS: was the path sound?

def task_completed(trace) -> bool:                  # OUTCOME: did it hit the GOAL? (binary judge)
    return llm_judge(goal=infer_goal(trace), final=trace.answer, steps=trace.steps)

def step_efficiency(actual, gold) -> float:
    return len(gold) / max(len(actual), 1)          # 1.0 = no wasted steps; < 1 = bloated / looping

# A right answer down a wasteful or risky path is a CORRUPT SUCCESS -> gate on outcome AND process.
ship = task_completed(trace) and process_ok["score"] and step_efficiency(actual, gold) >= 0.8

The same calibration discipline from L169 (Judge Biases & How to Calibrate) applies: validate your trajectory judge against human labels before you trust it. And whatever you use — reference match or LLM judge — gate on outcome and process together. A right answer with a 50%-efficiency, plan-drifting trajectory is not a pass.

Agent Failure Modes (Read From the Trace)

Trajectory evals exist to catch the failure modes that outcome alone hides. Knowing the catalog tells you what to look for in a trace (L177) and what your evals must score:

Wrong tool selection — calls an inappropriate tool (even if the final answer happens to be right).
Wrong arguments — right tool, wrong parameters (the \$300 refund). Caught only by comparing args.
Wrong order — does the right things in a risky sequence (reserve before check_inventory). Caught by strict.
Looping — repeats the same call unnecessarily. Caught by efficiency.
Plan drift — abandons its plan after a tool result arrives and wanders off.
Premature stop — quits before the goal is reached (low recall — a needed call is missing).
Hallucinated tool — calls a tool that doesn't exist, or with a fabricated signature.
Error compounding — an early mistake cascades through every later step.

Notice each failure maps to a specific metric or match mode — which is exactly why a single “did it succeed?” score is insufficient. “Correctness at the component level masks trajectory failures.”

🧪 Try It Yourself

Add trajectory evals to one real agent this week:

Capture trajectories. From your traces (L177), pull the ordered list of tool calls (with arguments) for ~20 representative runs.
Write gold references for the ones with a clear right path. For the open-ended ones, skip the gold and plan to use a judge.
Pick a match mode. Does order matter for this task? Use strict. Many valid orders? unordered. Start there, and always tool_args_match_mode="exact".
Compute the four numbers: outcome (task completed?), tool precision/recall, arg accuracy, and step efficiency (gold_len / actual_len).
Hunt for corrupt successes. Filter to runs where outcome = pass but efficiency < 0.8 or the match failed. Those are the fragile agents your outcome metric was hiding.

When you can say “92% task completion, but 18% of passes are corrupt successes — mostly looping on retrieval,” you've moved from “the agent seems fine” to an instrument that tells you exactly how to make it reliable.

Mental-Model Corrections

“Right answer = good agent.” It might be a corrupt success — a right answer via a wasteful, risky, or wrong-for-the-wrong-reason path. Judge outcome and process.
“Exact-match the trajectory to one gold path.” Agents have many valid paths — strict is often too brittle. Pick the mode (unordered/superset) or an LLM judge that matches how much flexibility is valid.
“Tool correctness = it called the tool.” Also the arguments. A right tool with a wrong arg (refund \$300) is a real failure — set tool_args_match_mode="exact".
“More tool calls = more thorough.” Efficiency matters — loops and retries cost money and latency and invite drift. Penalize wasted steps.
“Task completion is enough.” It's blunt — it says what failed, not where. Trajectory metrics localize the failure (which tool, which arg, which step).
“I need a gold trajectory for everything.” For open-ended tasks, use a reference-free LLM judge on the trajectory (reasoning, plan adherence, efficiency) — calibrated like any judge (L169).

Key Takeaways

Judge agents on two axes: outcome (did it hit the goal?) and trajectory (was the path sound?). The dangerous case is a corrupt success — a right answer down a bad path — which outcome-only eval ships blind.
Trajectory match has modes: strict (same tools, order, args — use when order matters), unordered (same calls, any order), superset (at least the gold tools). Pick by how much path-flexibility is valid; agents have many valid paths, so strict is often too brittle.
Tool correctness = selection + arguments. Summarize selection as precision (of called tools, % right) and recall (of needed tools, % covered); always compare args (a wrong-arg call is a real failure).
Step efficiency penalizes loops, retries, and wasted calls (gold_len / actual_len, or a judge) — extra steps cost money, latency, and reliability.
No single gold path? Use a reference-free LLM judge on the trajectory (reasoning quality, plan adherence, efficiency) — calibrated like any judge (L169). Gate on outcome AND process.
It all reads from the trace (L177). Next: the tools that capture traces and run these evals — LangSmith, Langfuse, Phoenix, Braintrust, Helicone (L179 (Observability Tools: LangSmith, Langfuse, Phoenix, Braintrust, Helicone)).