Prompt Caching and Reuse: Design Prompts That Save 90% on Token Costs at Scale

Every multi-step agent you ship does something expensive by default. On step 1 it pays full price for a 10k or 20k system prompt. On step 2 it sends that same prefix again. On step 26 it still pays for most of it, unless you designed the prompt for cache hits.

Prompt caching is not a toggle that makes your whole LLM invoice 90% cheaper. It is a prefix-match problem. Put stable content first, put dynamic content last, and keep the cached prefix byte-identical across requests. Do that on Anthropic or Gemini-class pricing and cached input can run at about 0.1x base (roughly 90% off that portion). Real total-bill savings usually land closer to 40-70%, because output tokens and the uncached tail still bill at full rate.

The $500 to $50 story is real when your bill is dominated by a large reusable prefix and hit rates stay high. It is not real when you mostly pay for long generations on short prompts.

What the model is actually reusing

On each request the model runs a prefill pass: it builds attention keys and values for every input token. Caching stores that prefill work for a matching prefix so later requests skip re-computing it.

The match rule is strict. One changing character in the "stable" region invalidates everything after it. That is why prompt structure is the cost control plane, not the provider checkbox.

Quality does not change when caching works correctly. Decode still runs normally. Same model, same sampling, same distribution of outputs. You cut prefill cost without stripping context or downgrading models.

The one structure that actually hits cache

Use this order on every provider:

``` [stable tools] [stable system: voice, policies, few-shots, product rules] [stable reference material reused across requests]