NOTES / EVALS AND OBSERVABILITY

Trajectories and final outcomes

How to judge the path an agent took and the answer it ended with, the difference between grounded and correct, and how to read the two together.

UPDATED 18 NOV 2026#evals#agents#trajectories

Evaluating routers, tools and skills graded single steps. This note grades what happens across the steps and what the user finally gets.

What a trajectory is

A trajectory is the sequence of actions an agent took on a task: receive the request, call lab lookup, read the result, call medication lookup, answer. Trajectory evaluation asks whether that path was appropriate.

Consider “Compare the patient’s current medications with their recent lab results.” A reasonable trajectory fetches medications, fetches labs, analyses, and responds. A problematic one fetches medications and responds. The answer may even read well, but the agent never looked at the labs.

What to check

  • Were the necessary actions performed?
  • Were irrelevant or repeated actions avoided?
  • Were tools called in an acceptable order?
  • Did the agent use each tool result correctly?
  • Did it stop once it had enough information?
  • Did it handle tool failures sensibly?

Exact sequence or acceptable behaviour

The hardest design question is how strict to be. Some requirements are about order: verify identity before retrieving a record. Others are not: it doesn’t matter whether medications or labs come first.

Diagram: four trajectories: two acceptable orderings, one that skips a required step, and one where the order itself is enforced

Diagram: my own.

A rigid exact-match evaluator would fail a perfectly good alternative ordering. Anthropic makes a related point: agents often find valid approaches the eval author didn’t anticipate, so tests that insist on one specific sequence of steps tend to be brittle1. Define the requirement at the right level: required actions and constraints where you can, exact order only where order truly matters.

def check_trajectory(
    actual: list[str],
    required: set[str],
    forbidden: set[str] = frozenset(),
    must_precede: list[tuple[str, str]] = (),
) -> dict:
    """Check a list of tool names against requirements, not one fixed path."""
    index = {name: i for i, name in enumerate(actual)}
    return {
        "required_actions_done": required <= set(actual),
        "no_forbidden_actions": not (forbidden & set(actual)),
        "order_respected": all(
            a in index and b in index and index[a] < index[b]
            for a, b in must_precede
        ),
        "no_repeated_calls": len(actual) == len(set(actual)),
    }

# Both orderings pass; skipping the lab call fails.
check_trajectory(
    ["get_lab_results", "get_medications", "respond"],
    required={"get_medications", "get_lab_results"},
)

The function takes the tool names in order, such as those read off a trace. Tune the strictness per case. For identity checks pass must_precede=[("verify_identity", "get_record")]; for the medication-and-lab comparison pass nothing at all.

Final outcomes

The final outcome is what the user actually receives, or what the agent accomplished. For an information assistant, the usual criteria are:

  • Correctness: are the claims accurate?
  • Completeness: are all parts of the request addressed?
  • Grounding: is each claim supported by evidence the agent had?
  • Relevance: does it answer the question asked?
  • Uncertainty: does it admit what is missing or unclear?
  • Format: does it follow the required structure?

Which ones you score is a product decision. If records show HbA1c of 6.4% in March, 5.8% in April and 5.7% in July, the reply “the latest is 5.7% in July, down from 6.4% in March” has the latest value and the overall change. Whether it must also mention April depends on how you defined completeness for this task.

Grounded is not the same as correct

These two are easy to confuse.

Diagram: an out-of-date retrieved record of 6.4% leads to an answer of 6.4%, which is grounded in the record but incorrect against the real current value of 5.7%

Diagram: my own; values are invented.

An answer is grounded when it is supported by the information the agent was given. It is correct when it matches the right reference. If the retrieved record is out of date, the agent can faithfully report it and still be wrong about the patient today. That is why a grounding judge alone isn’t enough, and why retrieval and source quality get evaluated in their own right.

When information is missing

If no HbA1c result exists, an acceptable answer is “I couldn’t find an HbA1c result in the available records.” A system must never be rewarded for producing a confident number when the evidence isn’t there. Build such cases into the eval set and score the honest refusal as a pass.

Reading outcome and trajectory together

Look at the two judgements side by side, because each can cover for the other.

Diagram: a two-by-two grid of outcome correct or incorrect against trajectory appropriate or inappropriate with what to investigate in each cell

Diagram: my own.

  • Correct and appropriate: the result you want.
  • Correct but inappropriate: the agent may have been right by chance, or wasted steps. Don’t celebrate yet.
  • Incorrect but appropriate: the plan was sound, so look at execution, interpretation or the final writing.
  • Incorrect and inappropriate: look at planning, routing and tool use.

A pass on the final answer alone would put the second row in the “fine” pile, and that is exactly where fragile agents hide. The matrix idea, along with the example, comes from the course2.

Sources

  1. [1]Demystifying evals for AI agents — Anthropic · Anthropic Engineering · 2026PARAPHRASED
  2. [2]Evaluating AI Agents (short course) — John Gilhuly and Aman Khan · DeepLearning.AI, with Arize AISTUDY SOURCE; EXAMPLES, CODE AND DIAGRAMS ARE MY OWN

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.