Prompt caching
Reuse the computed state of an identical prompt prefix to cut input cost and latency: how provider caches are priced, how to lay out prompts, and what breaks a hit.
What it is
When you send a prompt, the model first runs prefill over every input token, which builds the KV cache. If the next request begins with the same tokens, that work is identical. Prompt caching keeps the computed prefix so the provider can skip it, which lowers both cost and time to first token.
Diagram: my own.
This is a natural fit for agents, whose requests repeat the same tools, instructions and growing history every turn (The execution loop).
How providers do it
Anthropic lets you mark cache breakpoints with cache_control, and also supports automatic caching, in which the cache applies to the prompt prefix up to a point1. The docs state the pricing multipliers: a 5-minute cache write costs 1.25 times the base input price, a 1-hour write costs 2 times, and a cache read costs 0.1 times1. The default lifetime is 5 minutes, and it is refreshed each time the cached content is used2. Responses report cache_creation_input_tokens and cache_read_input_tokens so you can verify hits2. Up to four breakpoints are allowed per request, and there is a minimum prefix length that varies by model; shorter prefixes are processed normally without caching, so check the current table1.
Diagram: my own, from the multipliers in 1. Prices and multipliers change, so check the docs.
OpenAI introduced automatic prompt caching in 2024: no code change, a discount on recently seen input tokens, and a cached_tokens field in the usage data3. Its launch post described a 50 percent discount, and OpenAI’s cookbook says caching applies to prompts over 1,024 tokens and describes large latency reductions for long prompts6. Discount levels differ by model and have changed, so check the current pricing page.
Open-source engines do the same inside your own server. SGLang’s RadixAttention reuses the KV cache across calls that share prefixes4, and vLLM’s docs list prefix caching as a feature5.
Break-even
With Anthropic’s published multipliers, a 5-minute write costs 25 percent more than normal input, so if the same prefix is read once within the window you are already ahead: 1.25 + 0.1 is 1.35 against 2.0 for two uncached requests (my arithmetic from 1). A 1-hour write at 2x needs a couple more reads to pay off. Caching helps when a prefix is reused often and within its lifetime.
Design prompts for caching
- Put stable content first, changing content last. Order: tools, system prompt, reference documents, then conversation, then the new message.
- Keep the prefix byte-identical. A timestamp, a request ID or reordered tool list at the top defeats the cache. Serialise JSON and tool definitions deterministically.
- Append, do not edit history. Changing an earlier message invalidates everything after it. Prefer adding new messages over rewriting old ones.
- Place breakpoints at stable boundaries, such as after tools and system prompt, and after the last stable document.
- Match the lifetime to your call spacing. If calls are more than 5 minutes apart, a longer cache, or no cache, may suit.
- Keep settings that affect the prefix consistent. Anthropic’s guidance for troubleshooting misses points to checking that things like tool choice and image usage remain the same between calls2.
Tension with other techniques
- Compaction. Summarising or trimming history rewrites the prefix, so the next call pays a cache write. Compact in larger, less frequent steps (Compaction and summarisation).
- Dynamic tool loading. Changing the tool list each turn moves the prefix. Keep tool lists stable where you can (Tool design).
- Memory injection. Insert recalled memories after the stable prefix, not before it (Memory).
Measure it
Log cache read tokens, cache write tokens and uncached input tokens per request, and compute the hit rate. Watch for sudden drops after a prompt or tool change: they usually mean something at the top of the prompt started varying. Compare cost with and without caching on real traffic (Latency and cost).
Caching is not a security boundary. Check each provider’s data-retention and isolation terms if prompts contain sensitive data (Privacy and regulation).
Sources
- [1]Prompt caching — Anthropic docsPARAPHRASED
- [2]Prompt caching (earlier version of the docs) — Anthropic docsPARAPHRASED
- [3]Prompt Caching in the API — OpenAI · 2024PARAPHRASED
- [4]SGLang: Efficient Execution of Structured Language Model Programs — Zheng et al. · 2024PARAPHRASED
- [5]Welcome to vLLM — vLLM project docsPARAPHRASED
- [6]Prompt Caching 101 — OpenAI cookbookPARAPHRASED