NOTES / READING LIST AND GLOSSARY
Foundational papers
Eleven papers behind the topics on this site, each with a one-line takeaway in my own words and where it shows up in these notes.
UPDATED 18 NOV 2026#papers#reading
Diagram: my own timeline, by first arXiv submission.
Summaries below are mine, based on the abstracts and my reading; open the papers for the real detail. Order is chronological.
- Attention Is All You Need (2017). Introduces the Transformer, which drops recurrence for attention and trains faster in parallel.1 Background for LLM fundamentals.
- Retrieval-Augmented Generation (2020). Combines a generator with a retriever over external documents, so knowledge can be updated and inspected.2 See RAG overview.
- LoRA (2021). Adapts a large model by training small low-rank matrices instead of all weights.3 See LoRA and QLoRA.
- Chain-of-thought prompting (2022). Showing worked reasoning steps in the prompt improves performance on reasoning tasks for large models.4
- InstructGPT (2022). Fine-tunes with human feedback; labellers preferred a much smaller tuned model to a far larger base model.5 See preference tuning.
- ReAct (2022). Interleaves reasoning and actions so a model can use tools and update its plan.6 See ReAct and planning.
- Toolformer (2023). A model learns, in a self-supervised way, when to call simple APIs.7 See tool calling.
- Direct Preference Optimization (2023). Optimises preferences without a separate reward-model and RL loop.8
- Lost in the Middle (2023). Models use information at the start or end of a long input better than the middle.9 See long-context trade-offs.
- PagedAttention (2023). Manages the KV cache in pages to waste less memory when serving.10 See KV cache.
Findings in older papers may not hold for newer models; treat them as ideas to test with your own evals.
Sources
- [1]Attention Is All You Need — Vaswani et al. · 2017PARAPHRASED
- [2]Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al. · 2020PARAPHRASED
- [3]LoRA: Low-Rank Adaptation of Large Language Models — Hu et al. · 2021PARAPHRASED
- [4]Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Wei et al. · 2022PARAPHRASED
- [5]Training language models to follow instructions with human feedback — Ouyang et al. · 2022PARAPHRASED
- [6]ReAct: Synergizing Reasoning and Acting in Language Models — Yao et al. · 2022PARAPHRASED
- [7]Toolformer: Language Models Can Teach Themselves to Use Tools — Schick et al. · 2023PARAPHRASED
- [8]Direct Preference Optimization, your language model is secretly a reward model — Rafailov et al. · 2023PARAPHRASED
- [9]Lost in the Middle: How Language Models Use Long Contexts — Liu et al. · 2023PARAPHRASED
- [10]Efficient memory management for large language model serving with PagedAttention — Kwon et al. · 2023PARAPHRASED