Prompt Caching and Reuse: Design Prompts That Actually Hit the Cache at Scale

Every multi-step agent you ship does something expensive by default. On step 1 it pays full price for a 10k or 20k system prompt. On step 2 it sends that same prefix again. On step 26 it still pays for most of it, unless you designed the prompt for cache hits.

Prompt caching is not a toggle that makes your whole LLM invoice 90% cheaper. It is a prefix-match problem. Put stable content first, put dynamic content last, and keep the cached prefix byte-identical across requests. Do that on Anthropic or Gemini-class pricing and cached input can run at about 0.1x base (roughly 90% off that portion). Real total-bill savings usually land closer to 40-70%, because output tokens and the uncached tail still bill at full rate.

The $500 to $50 story is real when your bill is dominated by a large reusable prefix and hit rates stay high. It is not real when you mostly pay for long generations on short prompts.

What the model is actually reusing

On each request the model runs a prefill pass: it builds attention keys and values for every input token. Caching stores that prefill work for a matching prefix so later requests skip re-computing it.

The match rule is strict. One changing character in the "stable" region invalidates everything after it. That is why prompt structure is the cost control plane, not the provider checkbox.

Quality does not change when caching works correctly. Decode still runs normally. Same model, same sampling, same distribution of outputs. You cut prefill cost without stripping context or downgrading models.

The one structure that actually hits cache

Use this order on every provider:

``` [stable tools] [stable system: voice, policies, few-shots, product rules] [stable reference material reused across requests]

[conversation history so far, append-only] [dynamic: the new user message, retrieved chunks, timestamps, per-user data] ```

Everything above the dynamic line is the same bytes on every call. Everything below it is allowed to change. The order is not arbitrary on Anthropic either: its cache prefix is built from tools, then system, then messages, in that order, so a change in the tools block costs you everything after it.

Two rules make this work in practice. Nothing request-specific goes above the line, not even something small like today's date. And the conversation history only grows at the end. You never edit, reorder or summarize earlier turns in place, because each of those rewrites the prefix the next request is trying to match.

Five things that quietly break the prefix

None of these throw an error. The request succeeds, the answer looks fine, and you pay full input price on every call.

1. A timestamp or date in the system prompt. "Today is August 10, 2026, 14:02" at the top means the prefix changes every request. Put the date in the dynamic tail, or round it to the day and accept one cache write per day.
2. User data in the stable region. A greeting with the customer's name, their plan tier, or their account ID inside the system prompt gives every user their own prefix. Keep the system prompt identical for everyone and pass user facts in the message.
3. Non-deterministic serialization. Tool lists built from a dictionary, JSON with unsorted keys, or reference documents loaded in whatever order the filesystem returns them all produce different bytes for the same content. Sort and serialize them the same way every time.
4. Editing tools mid-session. On Anthropic, changing a tool's name, description or parameters invalidates the entire cache. Adding or removing images anywhere in the prompt, or changing tool_choice or thinking settings, invalidates the cached message blocks. Pick these settings at the start of a session and hold them.
5. Rewriting history. Trimming old turns, collapsing tool results, or swapping in a summary halfway through an agent run changes the prefix from that point on. If you have to compact, do it rarely and on purpose, and expect the next call to pay for a fresh write.

Provider by provider

The structure above is the same everywhere. What differs is how much control you get and what you have to set.

Anthropic. Caching is opt-in. You either mark individual content blocks with cache_control or set one cache_control at the top level of the request and let the API place the breakpoint on the last cacheable block. You get up to four breakpoints. The default cache lifetime is five minutes, refreshed at no extra cost each time the entry is read, and a one-hour lifetime is available. A cache write costs more than a normal input token, and the one-hour write costs more than the five-minute one, so caching a prefix you never reuse is a small loss, not a free option. Check cache_creation_input_tokens and cache_read_input_tokens in the response usage to see what actually happened.

OpenAI. Caching is on by default for supported models once the prompt passes a minimum length, and it matches on the longest identical prefix. The docs give the same advice as this post: stable instructions and shared reference material first, timestamps and user-specific details last. The prompt_cache_key parameter helps route related requests to the same cache on models before GPT-5.6. GPT-5.6 added explicit cache controls, and its cache writes bill above the uncached rate, so track both cached tokens and cache write tokens there. Cache hits show up in usage.input_tokens_details.cached_tokens.

Google Gemini. Implicit caching is on by default for Gemini 2.5 and newer models, above a minimum input size. Google's guidance for raising the hit rate is to put large, common content at the start of the prompt and to send requests with a similar prefix close together in time. For supported models you can also create an explicit cache object for a large document or corpus and reference it across requests until its TTL runs out.

Multi-step agents: move the breakpoint forward

Agents are where caching pays the most and where it breaks most often, because the prompt grows on every step.

The pattern is simple. Keep tools and the system prompt fixed for the whole run. Append each tool call and tool result to the end of the history. On every request, put the cache breakpoint on the newest block. Each step then reads the prefix the previous step wrote and only pays to write the new tail.

On Anthropic there is one detail to know. The API looks back a limited number of blocks (up to 20 positions per breakpoint) from where you place the breakpoint to find an earlier cache entry. A step that appends many content blocks at once can push the last write out of reach. If your agent produces bursts of tool results, add a second breakpoint earlier in the history so there is always an entry within range.

Audit your template with a prompt

Most cache breakers are easy to miss when you wrote the template yourself. Hand it to a model and ask for a stable versus dynamic map.

The Prompt:

You are reviewing an LLM request template for prompt cache hits.
The provider reuses cached work only for a byte-identical prefix.

Below is the template, with every variable shown as {{name}}.

For each line or block, label it STABLE (identical on every request)
or DYNAMIC (can change between requests). Then:
1. List every DYNAMIC item that appears before the last STABLE block.
   These are cache breakers.
2. For each one, say where it should move so the stable prefix stays
   unbroken.
3. Flag anything whose bytes could change without anyone meaning to:
   dates, IDs, unsorted lists, serialized objects, tool lists built
   at runtime.
4. Return the reordered template.

Do not rewrite wording. Only reorder and flag.

Template:
<paste template here>

Expected Output:

Line 1, "You are the support assistant for {{company_name}}": DYNAMIC if you serve more than one company. Cache breaker. Move the company name into the user message, or keep one template per company.

Line 3, "Current time: {{now}}": DYNAMIC. Cache breaker. Move it to the end, just before the user question.

Tool definitions, "{{tools}}": flagged. Built at runtime; confirm the list is sorted and serialized the same way on every request.

Policy text, few-shot examples, product reference: STABLE.

Reordered template: tools, policy text, examples, product reference, then company name, current time, retrieved chunks, user question.

Measure the hit rate, not the invoice

The monthly bill tells you too late and mixes everything together. Log the cache fields per request instead.

On Anthropic, total input is cache_read_input_tokens + cache_creation_input_tokens + input_tokens, so the share of input served from cache is the read count divided by that total. On OpenAI, compare cached_tokens to the request's input tokens. Track that share per endpoint and per agent step. When it drops, someone changed the stable region, and the diff of the last deploy usually tells you who.

Then be honest about what the number can buy you. A high hit rate cuts the cost of the cached input. It does nothing for output tokens or the uncached tail. If your bill is mostly long generations, caching is a smaller lever than shorter outputs or a cheaper model for the bulk steps.

What to ship this week

Pick your highest-volume endpoint. Move every dynamic value below the stable prefix, sort anything serialized, turn caching on where it is opt-in, and log the cache fields. Run it for a day and read the hit rate. That one number tells you whether your prompt is designed for the cache or just hoping for it.

Want hands-on training on prompt caching and cost-aware agent design for your team? Connect with Kief Studio on Discord or schedule a session.