RAG overview
Retrieval-augmented generation in one picture: index your documents, fetch the relevant pieces at question time, and have the model answer from them.
The problem RAG solves
A model’s weights hold what it saw in training. They are hard to update, cannot cite their sources, and do not include your private documents. The original RAG paper framed this as a limit of purely parametric models: they struggle to access and precisely manipulate knowledge, to give provenance for their answers, and to keep their world knowledge up to date1. Its answer was to combine a pretrained generator with a non-parametric memory, a dense index of documents reached through a neural retriever, so that knowledge can be revised and the accessed passages can be inspected1.
In today’s usage “RAG” usually means something simpler than the paper’s end-to-end training: fetch relevant text and put it in the prompt. The idea is the same.
The pipeline
Diagram: my own.
- Index time. Split documents into chunks (Chunking), compute representations (Embeddings and vector search, BM25), and store them with metadata.
- Query time. Turn the question into a search (Query rewriting), retrieve candidates, optionally fuse and rerank them (Hybrid search and reranking), and place the best passages in the prompt.
- Generation. The model answers from those passages, ideally citing them.
In the paper’s experiments, RAG models produced more specific, diverse and factual language than a parametric-only baseline on generation tasks1. That is a research result on older models; measure the effect on your own data.
Where it fails
Diagram: my own, aligned with the quality dimensions in 2.
The Ragas authors name three quality dimensions: whether the retriever finds relevant and focused passages, whether the model uses them faithfully, and the quality of the generation itself2. They also note that too-long contexts make models less effective at exploiting them, especially for information in the middle2, which links back to long-context trade-offs.
Most RAG problems are retrieval problems. Anthropic’s Contextual Retrieval write-up starts from this view: chunks often lose the context they had inside their document, so retrieval misses them3. Fix retrieval first, then look at the prompt.
When to use it
- Use RAG for private, changing or large knowledge, when you need citations, or when the answer lives in a few passages.
- Consider long context instead when the whole corpus fits and questions need global reasoning (Long-context trade-offs).
- Consider fine-tuning for style and behaviour, not for facts that change (When to fine-tune). These guidelines are mine, not from the sources.
Next: Chunking.
Sources
- [1]Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Patrick Lewis and co-authors · arXiv, NeurIPS 2020 · 2020PARAPHRASED
- [2]Ragas: Automated Evaluation of Retrieval Augmented Generation — Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert · arXiv · 2023PARAPHRASED
- [3]Introducing Contextual Retrieval — Anthropic · 2024PARAPHRASED