Skip to content
Search
TechniquesGlossary

Prompt Caching

Many requests share a long, fixed beginning: a system prompt, tool definitions, a reference document, a set of examples. With caching, the provider keeps the work it did on that prefix for a short time, and the next request that begins with exactly the same tokens reuses it. Cached input is billed at a discount. OpenAI's prompt caching guide says caching is on by default for supported models and that reuse requires the entire prefix to match. Anthropic's prompt caching docs let you mark where the cacheable prefix ends, with a five-minute default lifetime and a longer option at extra cost.

To benefit, put stable content first and anything that changes (the user's question, today's date, retrieved results) last. One changed character early in the prompt breaks the match for everything after it. Caching affects what you pay and how long you wait, while the model reads the same prompt either way. Qurtoo's article on prompt caching and reuse works through prompt layouts in detail.

Example

# Order a prompt so the cache can hit
1. Tool definitions            (same on every call)
2. System prompt and rules     (same on every call)
3. Reference documents         (same for this customer)
4. Few-shot examples           (same on every call)
--- everything above is the cacheable prefix ---
5. Conversation so far         (grows each turn)
6. Today's date and the new question (changes every call)

When to use it

  • Long system prompts or documents reused across many calls
  • Multi-turn chats and agent loops that resend the same history
  • High-volume pipelines where input tokens are most of the bill

When not to use it

  • Short prompts below the provider's minimum cacheable length
  • Calls spaced further apart than the cache lifetime
  • Prompts that put changing content (timestamps, user names) at the top