NOTES / EVALS AND OBSERVABILITY
Why evals come first
Why testing an LLM application is different from testing ordinary code, the three levels you can evaluate at, and the loop every later note plugs into.
Tests versus evals
An ordinary unit test pins down one right answer: assert add(2, 3) == 5. LLM applications break that habit in two ways. Several different answers can be correct, and a fluent answer can still be wrong.
Suppose a clinical-records assistant is asked for a patient’s latest HbA1c. It might say “The patient’s most recent HbA1c is 5.7%.” or “The latest recorded HbA1c was 5.7 percent.” Both carry the same fact. A string comparison would pass one and fail the other, so what you need is a check on the fact, not the wording.
Diagram: my own. All patient data in these notes is invented.
Anthropic’s working definition is a good place to start: an eval is a test for an AI system, where you give it an input and then apply grading logic to the output to measure success1. The grading logic is the part that has to be rethought.
Three things you can evaluate
“Evaluation” covers three different questions, and mixing them up is a common source of confusion.
Diagram: my own, using the model-versus-application distinction described in 2.
Model evals measure what the base model can do in general. Public benchmarks such as MMLU and HELM are of this kind2. Application evals ask a different question: does our pipeline do its task, on our data, for our users2. A model can score well on a benchmark and still fail your product, because your product feeds it the wrong context, calls the wrong tool or combines results badly.
Component evals sit one level below. They test a single part, such as the retriever or the router, so that a failure points at something you can fix. The rest of this section is mostly about application and component evals.
Why agents are harder
A chat completion is one input and one output. An agent runs a chain of decisions1:
- Interpret the request.
- Choose a tool.
- Write the tool’s arguments.
- Read the result.
- Decide whether to act again.
- Write the final answer.
Anthropic points out that agents call tools over many turns and change state as they go, so mistakes can propagate and compound1. Two consequences follow:
- A correct final answer does not prove every step was sound. The agent may have got there by luck or by wasteful detours.
- A wrong final answer could have started at any step.
So agent evaluation looks at the outcome and at the behaviour that produced it. Evaluating routers, tools and skills and Trajectories and final outcomes take that apart.
The loop
Everything in this section hangs off one loop. You collect examples, run the agent on them, evaluate the results, inspect the failures, change the system, and run the same examples again.
Diagram: my own.
The point is to replace “the answer looks fine to me” with repeatable evidence about what works, what fails and whether a change helped. Anthropic notes that writing evals early forces a team to state what success means, and that teams without them tend to find problems only in production, where fixing one failure can create another1. The course this section draws on follows the same arc: trace the agent, evaluate it systematically, then improve it3.
How the section fits together
| Step in the loop | Note |
|---|---|
| See what the agent did | Tracing and operational monitoring |
| Collect examples | Building eval sets |
| Decide how to grade | Evaluators and LLM-as-judge |
| Grade the parts of an agent | Routers, tools and skills and trajectories and outcomes |
| Compare changes | Experiments and eval-driven development |
| Keep learning from real use | Online evals and A/B tests |
Retrieval quality has its own metrics and lives in RAG and retrieval.
Sources
- [1]Demystifying evals for AI agents — Anthropic · Anthropic Engineering · 2026PARAPHRASED
- [2]The mirage of generic AI metrics — Hamel Husain · Decoding ML (guest post)PARAPHRASED
- [3]Evaluating AI Agents (short course) — John Gilhuly and Aman Khan · DeepLearning.AI, with Arize AISTUDY SOURCE; EXAMPLES AND DIAGRAMS ARE MY OWN