Evaluating Agents on Trajectories, Not Answers: A Benchmark Design Note

The scoring gap

Most agent benchmarks still grade like a multiple-choice exam: run the agent, compare its final answer to a reference, emit a score. That collapses an entire multi-step trajectory into a single bit. Two agents can both score 1.0 while one performed a clean, reversible operation and the other deleted a production table, retried four times, and arrived at the right answer by accident. If the benchmark cannot see the difference, neither can the team shipping the agent.

This is not a theoretical concern. Agent frameworks are increasingly used for tasks with real side effects: writing to databases, calling payment APIs, modifying infrastructure, sending messages. In those settings, the path is not an implementation detail. It is part of the product. A benchmark that ignores it optimises for the wrong thing.

Why final-answer grading fails

Final-answer grading assumes that the answer is the task. For question-answering that is roughly true. For agentic work it is not. Consider a research agent asked to reconcile two datasets. One trajectory reads both files, computes a diff, and writes a summary. Another reads the files, writes an intermediate file to the wrong directory, retries with a corrected path, reads a stale cached copy, and finally produces the same summary. Same output. Different reliability, different cost, different blast radius.

Three properties that matter in production are invisible to output-only scoring:

• Cost. Token usage, tool calls, and wall-clock latency. An agent that solves a task in one tool call and another that solves it in nine are not equivalent products, even if the answers match.
• Side effects. Every external system touched, and in what terminal state. A benchmark that does not record this cannot distinguish a read-only agent from one that mutated state and then cleaned up.
• Reproducibility. Whether the same agent on the same task produces the same trajectory across seeds. A single successful run is an anecdote, not a measurement.

What to record

The unit of evaluation should be the trajectory, not the answer. That means instrumenting the agent to emit a structured action trace. At minimum, each step should record:

• The tool name and its full arguments, not just the name. write_file is not informative; write_file(path="/etc/hosts", mode="append") is.
• The observation returned by the tool, including error payloads.
• Timestamps and token counts per step.
• The terminal state of every external system the run touched: files created or modified, rows inserted, messages sent, resources provisioned.

A minimal trace schema might look like this:

{
  "run_id": "...",
  "task_id": "...",
  "seed": 7,
  "steps": [
    {
      "index": 0,
      "tool": "read_file",
      "args": {"path": "data/input.csv"},
      "observation": {"status": "ok", "bytes": 4096},
      "tokens_in": 812,
      "tokens_out": 44
    },
    {
      "index": 1,
      "tool": "write_file",
      "args": {"path": "out/summary.md", "mode": "create"},
      "observation": {"status": "ok"},
      "tokens_in": 921,
      "tokens_out": 130
    }
  ],
  "terminal_state": {
    "files_created": ["out/summary.md"],
    "files_modified": [],
    "external_calls": []
  },
  "retries": 0,
  "backtracks": 0
}

The exact schema matters less than the discipline: arguments, observations, and terminal state. Without arguments, you cannot audit intent. Without terminal state, you cannot audit consequence.

Retry and backtrack counts are a useful proxy for brittleness. An agent that reaches the correct answer after three retries is more fragile than one that reaches it directly, and much more likely to fail under distribution shift. These counts should be reported alongside the outcome, not hidden inside it.

Grading trajectories

Once trajectories are recorded, scoring becomes a vector rather than a scalar. Four dimensions cover most production concerns:

1. Outcome. Did the agent achieve the task's success condition? This is the familiar binary or graded score.
2. Path validity. Did the trajectory respect the constraints of the task? Were the tools used appropriate, were arguments well-formed, were intermediate steps coherent?
3. Side-effect hygiene. Were unintended external systems left untouched? Were intended changes applied exactly once, idempotently, in the correct order?
4. Cost. Tokens, latency, and tool calls, normalised against a reference trajectory or a budget.

Reporting a vector is more informative than reporting a scalar, but it complicates leaderboards. One honest trade-off: vector scores are harder to rank. A team that cares about side effects and a team that cares about latency will weight the dimensions differently. The benchmark should expose the components and let consumers aggregate, rather than baking in a single weighting that pretends to be universal.

A second trade-off: trajectory grading requires instrumented environments. You cannot record terminal state for arbitrary external systems without either sandboxing them or building adapters. That is real engineering cost, and it is why many benchmarks avoid it.

Held-out environments and variance

Trajectory grading only works if the agent cannot have memorised the fixture. Held-out environments are the mechanism. The benchmark should generate or reserve task instances that the agent has never seen, ideally with randomised parameters: file names, directory layouts, API responses, and initial states. If the agent has seen the fixture during training or tuning, a high score measures recall, not capability.

Variance reporting is equally important. A single run is not a measurement. Agents are stochastic systems: sampling temperature, tool-call ordering, and environment timing all introduce variance. The benchmark should run each task across multiple seeds and report the distribution of scores, not just the mean. A high mean with a long tail of failures is a different product from a slightly lower mean with tight variance.

The benchmarking ecosystem is maturing in this direction. Coverage of new evaluation efforts, such as Vals' work on AI benchmarking, reflects a broader recognition that evaluating agents is not the same as evaluating models. Output-only leaderboards are being supplemented by process-aware evaluation, and for good reason.

A concrete mechanism: the sandboxed replay harness

One practical way to implement trajectory grading is a sandboxed replay harness. The harness provisions a fresh environment per run: a container with a filesystem, a mock API server, and a network policy that blocks or records outbound calls. The agent runs inside it. On completion, the harness snapshots the environment, diffs it against the initial state, and emits both the action trace and the terminal-state diff.

Because the environment is ephemeral, side effects are observable and contained. Because the API server is mock, responses can be randomised per seed. Because the network policy is explicit, unintended external calls are recorded rather than silently permitted. The harness then scores the run on the vector described above.

This is more work than running an agent against a static answer key. It is also the minimum required to make claims about agent reliability that survive contact with production.

What to report

A trajectory-aware benchmark report should include, per task:

• Outcome score, with the success condition stated explicitly.
• Path validity flags, with the specific violations enumerated.
• Side-effect diff, summarised as counts of created, modified, and deleted resources.
• Cost, broken down by tokens and tool calls.
• Retry and backtrack counts.
• Variance across seeds, reported as a distribution rather than a single number.

Aggregated across tasks, this gives a profile rather than a rank. Profiles are harder to market. They are also more useful for deciding whether an agent is safe to deploy.

Conclusion

Benchmarks that score only final outputs systematically reward agents that reach correct answers through unsafe or unreproducible paths. The fix is not a better answer key. It is a different unit of analysis: the trajectory, with arguments, observations, terminal state, and cost. Score on a vector. Use held-out environments. Report variance. Treat a single run as an anecdote.

The trade-off is real: trajectory grading is more expensive to build and harder to summarise. But the alternative is a leaderboard that cannot distinguish a careful agent from a lucky one, and a deployment decision made on the wrong signal.

---

Cover photo: Brett Sayles / Pexels.