NOTES / PROMPTING AND CONTEXT ENGINEERING

Long-context trade-offs

When to put everything in a huge window and when to retrieve instead: quality, latency and cost all move with length, so decide by measuring.

UPDATED 18 NOV 2026#long-context#retrieval#cost#latency

The temptation

Windows now reach hundreds of thousands or millions of tokens. It is tempting to paste in the whole corpus and ask. Sometimes that is exactly right. Often it is slower, pricier and less accurate than a smaller, better-chosen context.

What goes wrong at length

  • Position effects. In multi-document question answering, Liu et al. found performance fell when the relevant passage was in the middle of the input, compared with the start or end, and that accuracy dropped as context lengthened, even for long-context models1.
  • Non-uniform degradation. Chroma reported that reliability declines as inputs grow, in uneven and sometimes surprising ways, and that similar-looking distractors hurt more2.
  • Cost and latency. More input tokens cost more and take longer to process. See Latency and cost and KV cache.

Models have improved since these studies, so treat them as warnings to measure, not fixed numbers.

Stuff or retrieve?

Diagram: stuff everything in versus retrieve what is needed, with a note to measure both on your own eval set

Diagram: my own.

Situation Lean toward
One document or a small, stable set; tasks need global reasoning across it Long context
Large or changing corpus; questions touch a few passages Retrieval (RAG overview)
Same big prefix reused across many calls Long context plus prompt caching
Need citations to exact sources Retrieval, or long context with quoting
Agent exploring a codebase or files Just-in-time lookup rather than preloading4

That table is my synthesis, not a rule from the sources.

If you do use long context

Anthropic’s guide gives specific advice for inputs of 20k tokens and above. Put long documents near the top of the prompt, above the question, and wrap each document in tags with metadata. I also recommend placing the question after the documents and, for long inputs, asking the model to quote the relevant passages before answering. I am relying on summaries of the guide for those two points rather than its full text, so check the current page3.

Diagram: documents first in tags, the question after, optionally ask for quotes first

Diagram: my own, following the advice in 3.

Also:

  • Trim what is not needed. Even with a big window, shorter inputs are cheaper and often better4.
  • Evaluate at your real lengths. Build an eval with the relevant fact placed at different positions, and compare stuffed against retrieved.
  • Cache the stable prefix. Order the prompt so unchanging content comes first.

Next: Compaction and summarisation.

Sources

  1. [1]Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu and co-authors · arXiv, TACL · 2023PARAPHRASED
  2. [2]Context Rot: How Increasing Input Tokens Impacts LLM Performance — Kelly Hong, Anton Troynikov, Jeff Huber · Chroma · 2025PARAPHRASED
  3. [3]Prompting best practices — Anthropic, Claude Platform docsPARAPHRASED
  4. [4]Effective context engineering for AI agents — Anthropic Engineering · 2025PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.