NOTES / PART 3 · RUNNING IN PRODUCTION

08Evals and observability

Knowing whether a system works, and why it failed when it doesn't.

THIS SECTION COVERS

Why evals matter, tracing, building eval sets, evaluators and LLM-as-judge, evaluating agent components and trajectories, experiments, benchmarks, and online evals.

Related: Safety and guardrails · RAG and retrieval

TOPICS · 8/9 WRITTEN

  1. 08.1Why evals come firstWhy testing an LLM application is different from testing ordinary code, the three levels you can evaluate at, and the loop every later note plugs into.
  2. 08.2Tracing and operational monitoringHow to record what an agent actually did, using traces and spans, how to instrument it, how to read a trace when an answer is wrong, and which operational signals to watch.
  3. 08.3Building eval setsWhat an evaluation case contains, where cases come from, why you should start from error analysis, how to cover more than the easy cases, and how to keep a held-out set honest.
  4. 08.4Evaluators and LLM-as-judgeThe three ways to grade an output (code, an LLM judge, a human), how to choose between them, how to write and calibrate a judge, and the known ways judges go wrong.
  5. 08.5Evaluating routers, tools and skillsBreak an agent into components, then evaluate each one separately: the router that picks a path, the tool calls and their arguments, and the skills that do the work.
  6. 08.6Trajectories and final outcomesHow to judge the path an agent took and the answer it ended with, the difference between grounded and correct, and how to read the two together.
  7. 08.7Experiments and eval-driven developmentTurning scores into decisions: run versions of your agent against the same dataset, compare them case by case, fix one thing at a time, and guard against regressions.
  8. 08.8Benchmarks and their limitsPLANNED
  9. 08.9Online evals and A/B testsEvaluating an agent on live traffic: what to score, how to sample, how production failures flow back into your eval set, and where A/B tests fit.