NOTES / PROMPTING AND CONTEXT ENGINEERING
The context window as a budget
Everything the model sees competes for one finite window and one finite attention budget. Treat tokens as a scarce resource and keep only high-signal ones.
One window, many claimants
The context window is everything the model can consider in a call: system prompt, tool definitions, conversation history, retrieved documents, tool results, and room for the answer. All of it shares one token limit.
Diagram: my own. The proportions are illustrative.
In an agent, history and tool results grow every turn, so the budget is spent faster than people expect. Count tokens with the target model’s own tokenizer, since counts differ between models (Tokens and tokenisation).
A second budget: attention
Fitting is not the only constraint. Anthropic frames context as a finite resource with diminishing returns, and says models have an “attention budget” that gets stretched as context grows, because every token attends to every other token1. Their stated goal is the smallest set of high-signal tokens that makes the desired outcome likely1.
Research backs the intuition. Chroma tested many models and reported that models do not use their context uniformly: performance becomes increasingly unreliable as input length grows, even on simple tasks, and distractors make it worse2. Liu et al. found accuracy depends on where the relevant information sits, best at the start or end and worst in the middle3.
Diagram: my own and illustrative only. Real curves differ by model and task; measure yours.
Spending the budget well
My working rules, built on the sources above:
- Audit what is in the window. Log the full prompt for a sample of calls and see what actually occupies it. It is often mostly stale tool output.
- Keep tools few and clear. Anthropic notes that overlapping tools make it harder for the model, and for humans, to decide which to use1. See Tool design.
- Load just in time. Instead of pre-loading everything, keep lightweight references such as file paths or queries and fetch details when needed; Anthropic describes this as retrieving context dynamically at runtime1.
- Return small tool results. Trim, paginate or summarise what tools hand back.
- Prune and compact as you go. See Compaction and summarisation.
- Isolate with sub-agents when a side task would flood the main context. See Multi-agent systems.
- Reserve room for output. Reasoning and long answers need tokens too.
Next: Long-context trade-offs.
Sources
- [1]Effective context engineering for AI agents — Anthropic Engineering · 2025PARAPHRASED
- [2]Context Rot: How Increasing Input Tokens Impacts LLM Performance — Kelly Hong, Anton Troynikov, Jeff Huber · Chroma · 2025PARAPHRASED
- [3]Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu and co-authors · arXiv, TACL · 2023PARAPHRASED