NOTES / EVALS AND OBSERVABILITY
Evaluators and LLM-as-judge
The three ways to grade an output (code, an LLM judge, a human), how to choose between them, how to write and calibrate a judge, and the known ways judges go wrong.
What an evaluator is
An evaluator is any method that decides whether an output, or a piece of behaviour, meets a requirement. There are three families: code-based checks, LLM judges and human review2, and the course behind these notes teaches the same three4. The skill is matching each requirement to the cheapest evaluator that can judge it reliably.
| Family | Strength | Limitation |
|---|---|---|
| Code | Fast, repeatable, easy to explain | Cannot see meaning; may fail an equivalent answer |
| LLM judge | Handles language and semantics at scale | Can be wrong or inconsistent; needs checking |
| Human | Handles nuance; sets the reference standard | Slow, costly, and people disagree |
Code-based evaluators
Use code when the requirement is explicit and checkable. Examples for a clinical-records assistant: the output is valid JSON, the agent chose the expected tool, a required field is present, a number is within tolerance, the patient ID in the tool call matches the one requested.
def correct_tool(actual_tool: str, expected_tool: str) -> bool:
return actual_tool == expected_tool
def within_tolerance(actual: float, expected: float, tol: float = 0.05) -> bool:
return abs(actual - expected) <= tol
Code evaluators are the backbone of a good suite, because they cost almost nothing to run and never wobble. Their weakness is meaning. “5.7%” and “5.7 percent” are the same fact, and a rule has to be written to know it. If you find yourself writing ever-more-elaborate string rules, that is the signal to consider a judge.
LLM-as-judge
An LLM judge grades an output against instructions you write. It suits criteria that are about language, such as whether a summary is supported by the records it was given.
The most reliable pattern is one criterion per judge, with explicit pass and fail conditions and a structured result. A vague prompt such as “Is this answer good?” mixes concision, politeness, correctness and completeness into one muddle. A narrow one does not:
You are checking whether an answer is supported by the supplied records.
PASS if every clinical fact in the answer is supported by the records
and the answer adds no diagnosis or causal claim the records do not state.
FAIL if any clinical fact is not supported by the records.
Return JSON: {"verdict": "PASS" | "FAIL",
"reason": "<one or two sentences>",
"unsupported_claims": ["..."]}
Records:
{records}
Answer:
{answer}
That judge tests grounding and nothing else. Correctness against a reference and completeness get their own judges or code checks. Keeping them separate means a FAIL tells you what kind of failure occurred.
Choosing an evaluator
Diagram: my own.
| Requirement | Start with |
|---|---|
| Valid JSON, right tool, right patient ID | Code |
| Numeric value matches the expected fact | Code |
| Answer is supported by the source records | LLM judge, checked against human labels |
| Clinical interpretation is appropriate | Qualified human review, perhaps with a judge to prioritise |
| Reads clearly | Human review or a calibrated judge |
Husain’s warning about generic metrics applies here. A built-in score for something like “helpfulness” is not a verdict; at best it is a way to surface traces for a person to read, a starting point for investigation rather than its conclusion3. Prefer criteria that came from your own error analysis.
Judges make mistakes
LLM judges are useful but not neutral. Zheng and colleagues studied the approach directly and documented several biases: position bias (favouring an answer because of where it appears), verbosity bias (favouring longer answers), self-enhancement bias (favouring a model’s own style) and limited reasoning ability1. They also found that strong judges agreed with human preferences more than 80% of the time, about the level at which humans agree with each other1, and proposed ways to reduce some biases1. Both findings are true at once: judges are good enough to be worth using, and biased enough to need checking.
Practical consequences:
- When comparing two answers, run the comparison in both orders.
- Be wary of rewarding length; ask for specific criteria, not overall impressions.
- Consider using a different model for judging than for generating.
- Treat the judge’s explanation as a clue. It is not proof that the verdict was right.
Calibrating a judge
A judge is only as good as its agreement with the judgement you actually care about. Calibrate it against human labels.
Diagram: my own.
- Collect examples that people have labelled.
- Run the judge on the same examples.
- Compare verdicts and read every disagreement.
- Clarify ambiguous wording and add representative examples to the rubric.
- Re-check on fresh labelled examples that the rubric was not tuned on.
Count the two kinds of error separately.
Diagram: my own.
A false pass approves a bad answer; a false fail rejects a good one. Which matters more depends on the product. For a clinical assistant, silently passing an unsupported claim is usually the worse mistake, so tune the judge to be strict there and accept some extra false failures that a person can review.
Human annotation
People are the reference standard for nuance: is a summary medically accurate, does it omit a finding that matters, is a recommendation suitable for the context. Review is slow, costly and subject to disagreement, so write annotation guidelines, use domain experts where the judgement needs them, and measure how often reviewers agree. Human labels do double duty, as ground truth for the judgement itself and as the calibration set for your judge.
Sources
- [1]Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Lianmin Zheng, Wei-Lin Chiang, Ying Sheng and others · NeurIPS 2023 (arXiv:2306.05685) · 2023PARAPHRASED
- [2]Demystifying evals for AI agents — Anthropic · Anthropic Engineering · 2026PARAPHRASED
- [3]The mirage of generic AI metrics — Hamel Husain · Decoding ML (guest post)PARAPHRASED
- [4]Evaluating AI Agents (short course) — John Gilhuly and Aman Khan · DeepLearning.AI, with Arize AISTUDY SOURCE; EXAMPLES AND DIAGRAMS ARE MY OWN