NOTES / RAG AND RETRIEVAL

Measuring retrieval

How to tell whether your RAG system works: score retrieval, grounding and the final answer separately, using real questions and a few well-chosen metrics.

UPDATED 18 NOV 2026#rag#evaluation#metrics#ragas

Measure in layers

A RAG answer can be wrong because retrieval missed, because the context was noisy, or because the model misused good context. One end-to-end score cannot tell you which. Evaluate each layer.

Diagram: three layers: retrieval, generation, and end to end, each with its own metrics

Diagram: my own.

The Ragas paper makes the same point: RAG evaluation must consider the retriever’s ability to find relevant and focused passages, the model’s ability to use them, and the quality of generation, and automated evaluation is paramount because performance depends on the retriever, corpus, model and prompt together1.

Layer 1: retrieval

You need a set of real questions, each paired with the passages that contain the answer. Then ask whether those passages came back.

  • Recall@k. The share of questions whose relevant passage appears in the top k. Anthropic’s “retrieval failure rate” for the top 20 chunks is essentially the complement of this4.
  • Precision and ranking quality. How much of what you retrieved is relevant and how high the good items rank. Metrics such as nDCG@10, MAP and precision@k are standard in retrieval benchmarks like BEIR3. BEIR itself combines 18 datasets to test generalisation, and found BM25 a robust baseline3.
  • Token-level measures. Chroma argues that document-level metrics hide the effect of chunking, and uses token-level precision, recall and intersection-over-union against the answering excerpts2.

Keep k tied to what your prompt can hold. High recall at k = 200 is useless if you can only include five passages.

Layer 2: generation

Ragas defines three reference-free quality aspects, each measured by prompting a language model1:

Diagram: context relevance, faithfulness and answer relevance

Diagram: my own, after the three aspects in 1.

  • Faithfulness: the answer’s claims can be inferred from the retrieved context. This catches hallucination despite good retrieval.
  • Answer relevance: the answer actually addresses the question.
  • Context relevance: the context is focused, with little irrelevant text. This matters for cost and because models use long, padded contexts less well.

The authors validated these metrics on a dataset called WikiEval and reported strong agreement with human judgements for faithfulness and answer relevance1. LLM judges are useful but imperfect; spot-check them against human labels and see Why evals.

Layer 3: end to end

Does the user succeed? Track task success, citation correctness, abstention when the answer is not in the corpus, latency and cost per query. Include unanswerable questions so the system is rewarded for saying it does not know.

Building the question set

  1. Sample real queries from logs when you have them; otherwise write them with domain experts.
  2. Mark the supporting passages. A model can draft candidates, but a human should verify.
  3. Cover easy, hard, multi-hop, exact-identifier and unanswerable cases.
  4. Keep a held-out slice, and do not tune on it. See Building eval sets.
  5. Re-run after every change to chunking, embeddings, fusion or prompts, and watch production with tracing.

Public benchmark scores tell you little about your corpus. Your own labelled questions are the measurement that matters.

Sources

  1. [1]Ragas: Automated Evaluation of Retrieval Augmented Generation — Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert · arXiv · 2023PARAPHRASED
  2. [2]Evaluating Chunking Strategies for Retrieval — Brandon Smith, Anton Troynikov · Chroma · 2024PARAPHRASED
  3. [3]BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Nandan Thakur and co-authors · arXiv, NeurIPS 2021 · 2021PARAPHRASED
  4. [4]Introducing Contextual Retrieval — Anthropic · 2024PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.