NOTES / EVALS AND OBSERVABILITY
Building eval sets
What an evaluation case contains, where cases come from, why you should start from error analysis, how to cover more than the easy cases, and how to keep a held-out set honest.
What a case contains
An evaluation dataset is a set of examples you run the agent on, the same way every time. One case holds an input, whatever context the agent needs, and a description of what a correct result looks like.
Diagram: my own. The patient ID and values are invented.
The important design choice is expected_facts. It records what must be true of a good answer (value 5.7, unit percent, dated 1 July 2026) and not the sentence that says so. That lets a code check or a judge accept any correct wording, which is the whole problem described in Why evals come first.
Tags look minor but pay off later. They let you report scores by category, so a weak spot isn’t averaged away.
Where cases come from
Most useful sets mix several sources.
Diagram: my own.
| Source | Good for | Watch out for |
|---|---|---|
| Written from requirements | Behaviour you know must work | Reflects what you imagined, not what users do |
| Production traces | Realistic phrasing and mess | Privacy: strip or avoid sensitive data |
| Synthetic cases | Filling gaps cheaply | Treat as unverified until a person has reviewed them |
| Past failures | Preventing regressions | Skews towards problems you have already seen |
| Expert-labelled examples | Ground truth for judgement calls | Slow and costly; needs clear guidelines |
Phoenix, for one, supports versioned datasets of examples for experiments and evaluation3. Whatever tool you use, record where each case came from. It helps when you later ask why the set over-represents one kind of question.
Start from error analysis
It is tempting to begin by writing a rubric. Hamel Husain argues against that: a common mistake is writing the rubric before looking at any data, and the right first step is error analysis, where you read and annotate real traces1. It is the most human-intensive part of evals, and also the part that tells you what to test.
Diagram: my own, following the workflow described in 1.
The practical version is simple. Read a batch of real traces. Write a short note on each thing that went wrong. Group the notes into recurring categories. Each category becomes cases in your set and, where possible, a check. Categories come out of the data, so they match your product rather than a generic list of qualities.
Cover more than the easy cases
A set made only of straightforward examples gives a flattering and misleading picture. For a lab-lookup skill, a better set includes:
| Case | What it tests |
|---|---|
| One result exists | Basic retrieval |
| Several results, unordered | Picking the latest |
| No results | Saying so rather than inventing a value |
| Different date formats | Date handling |
| A similar test name (HbA1c versus glucose) | Choosing the right test |
| Tool returns an error | Handling failure honestly |
| Patient ID missing | Asking, not guessing |
Anthropic notes that writing evals early forces you to define what success means2. These rows are that definition made concrete, including what “success” is when the data is missing or the tool fails.
Development and held-out sets
Split the set. Use the development part freely while you tune prompts, tools and code. Keep the held-out part for checking, and look at it rarely.
The reason is the same as in machine learning generally. If you tune against every example, each fix quietly bakes in knowledge of those exact cases. Scores rise, but you can no longer tell whether the system got better or only got used to your test. A held-out set that you have not tuned against is the only honest check that improvements generalise.
When a held-out case exposes a real failure, you can promote it into the development set and write a fresh held-out case to replace it.
Keeping the set healthy
- Version it. You need to know which dataset produced which score.
- Tag every case by category so results can be sliced.
- Record provenance: authored, from production, synthetic, or labelled by whom.
- Review synthetic cases before trusting them as references.
- Retire stale cases when requirements change, and say why.
- Handle sensitive data deliberately if cases come from real traces.
Start small and reviewed rather than large and unchecked. A few dozen well-understood cases that you have actually read beat thousands you haven’t. Grow the set as online monitoring surfaces new kinds of failure. The case fields and coverage table here are my own adaptation of examples from the course4.
Sources
- [1]Automating Error Analysis — Hamel Husain · hamel.dev · 2026PARAPHRASED
- [2]Demystifying evals for AI agents — Anthropic · Anthropic Engineering · 2026PARAPHRASED
- [3]Arize Phoenix documentation — Arize AI · Arize AIPARAPHRASED
- [4]Evaluating AI Agents (short course) — John Gilhuly and Aman Khan · DeepLearning.AI, with Arize AISTUDY SOURCE; EXAMPLES AND DIAGRAMS ARE MY OWN