Prompt Caching for LLM APIs: Cut Costs by 90% in Python (2026)

A RAG chatbot with a 3,000-token system prompt, a few-shot example block, and a retrieved-context section sends roughly the same 4,000 tokens on every single request — and pays full input price for all of them, thousands of times a day. The user's actual question is 30 tokens. You are buying the same 4,000 tokens over and over.

Prompt caching fixes exactly this. The provider stores the processed prefix of your prompt and reuses it on subsequent calls, billing those tokens at a fraction of the input rate. This guide shows how to make it actually happen: where the breakpoints go, how to verify you are getting hits instead of assuming you are, and where provider caching stops being enough.

1. What Prompt Caching Actually Does

A transformer processes tokens left to right, so the internal state computed for the first N tokens does not depend on anything after them. If two requests share an identical first N tokens, that computation is identical and can be reused. That shared beginning is called the prefix.

When you get a cache hit, you pay a discounted rate for the cached input tokens instead of the full input rate; you still pay full price for everything after the cached prefix. Three properties matter in practice:

• Exact prefix matching. A single changed character anywhere in the prefix invalidates everything from that point onward. Not "similar" — identical.
• Caches expire. Anthropic's ephemeral cache has a 5-minute TTL that refreshes on each hit. OpenAI's automatic caching drops idle prefixes after a short window (longer during off-peak). Cold traffic gets no benefit.
• There is a write cost. The first request that populates the cache pays a premium over the normal input rate. Caching is only worth it when a prefix is read more times than it is written.

That last point is the whole economics of it. With Anthropic's pricing, a cache write costs 1.25× the base input rate and a cache read costs 0.1×. Break even on the second read; by the tenth read you are paying about a tenth of what you would have. Check each provider's pricing page for current numbers — the multipliers move, the mechanism does not.

2. Does Your Prompt Qualify?

Before writing any code, work out whether your workload has a cacheable shape. Caching wins when a large static prefix is reused across many requests within minutes:

There is also a minimum length. Prompts shorter than roughly 1,024 tokens are not cached (2,048 for Anthropic's Haiku-class models). Short prompts have nothing worth caching anyway.

3. Anthropic: Explicit Cache Breakpoints

Anthropic gives you manual control. You mark a content block with cache_control and everything up to and including that block becomes the cached prefix. You can set up to four breakpoints.

import anthropic

client = anthropic.Anthropic()

SYSTEM = (open("system_prompt.md").read() + "\n\n" +
          open("few_shot_examples.md").read())

resp = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": SYSTEM,
        "cache_control": {"type": "ephemeral"},
    }],
    messages=[{"role": "user", "content": user_question}],
)

u = resp.usage
print(u.cache_creation_input_tokens, u.cache_read_input_tokens, u.input_tokens)

Read the usage object on every call during development. cache_creation_input_tokens is the write (billed at the premium), cache_read_input_tokens is the hit (billed at the discount), and input_tokens is everything that missed. If you see creation tokens on every request and never any read tokens, your prefix is changing between calls and you are paying more than if you had not cached at all.

When the conversation grows, move the breakpoint rather than re-sending everything. Mark the newest turn with cache_control and drop the marker from earlier ones — each marker you set creates a separate cache entry with its own write cost:

messages = [
    {"role": "user", "content": long_document},          # stable
    {"role": "assistant", "content": first_answer},
    {"role": "user", "content": second_question,
     "cache_control": {"type": "ephemeral"}},            # extend cache to here
]

Four breakpoints is plenty. The usual pattern is: system prompt and tools, then the conversation up to the last turn. More markers means more writes, not more hits.

4. OpenAI: Automatic Caching

OpenAI requires no code changes. Prompts over about 1,024 tokens are hashed in 128-token increments starting from the beginning, and matching prefixes are served from cache automatically. Cached input tokens bill at a discount — roughly 50% for most current models, more for some — and there is no write premium.

Your job is therefore pure prompt hygiene: make sure the shared content is at the front and byte-identical across requests. Then confirm it in the usage object:

from openai import OpenAI

client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},   # must be identical every call
        {"role": "user", "content": user_question},
    ],
)

d = resp.usage.prompt_tokens_details
print(d.cached_tokens, "/", resp.usage.prompt_tokens)

Note the failure mode this creates: because caching is automatic, you can be silently missing it and never know. Log cached_tokens / prompt_tokens as a metric from day one. A hit rate under 20% on a workload you expected to cache well almost always means something in the prefix is varying.

5. Measure Your Hit Rate First

Do not add caching on faith. Wrap the call, log the token breakdown, and look at the ratio before you change anything else:

import logging, functools

log = logging.getLogger("llm")

def tracked(fn):
    @functools.wraps(fn)
    def wrapper(*a, **kw):
        resp = fn(*a, **kw)
        u = resp.usage
        cached = getattr(getattr(u, "prompt_tokens_details", None), "cached_tokens", None) \
                 or getattr(u, "cache_read_input_tokens", 0) or 0
        total = getattr(u, "prompt_tokens", None) or (
            u.input_tokens + u.cache_creation_input_tokens + u.cache_read_input_tokens)
        log.info("llm.tokens cached=%d/%d (%.0f%%)", cached, total, 100 * cached / max(total, 1))
        return resp
    return wrapper

ask = tracked(client.chat.completions.create)

Emit that as a counter, not just a log line. Cached-token ratio is the single metric that tells you whether your prompt architecture is healthy: it drops to zero the moment someone interpolates a timestamp into the system prompt, and you find out the same day instead of when the invoice arrives.

6. Prompt Ordering: The 90% Rule

Caching rewards stable content at the front and volatile content at the back. Most prompts are written in the opposite order, which is why most first attempts at caching produce a 0% hit rate. Reorder strictly:

1. System prompt — static, never interpolated.
2. Tool and function schemas — static, and often large. Huge win in agent loops.
3. Few-shot examples — static, and usually the biggest single block.
4. Retrieved context — semi-stable. If you retrieve top-k chunks, sort them by a stable key (document ID, chunk index) — never leave them in score order, which shuffles on every query.
5. User question — always last.

Then audit the static blocks for hidden volatility. These all silently break your prefix:

• A timestamp, datetime.now(), or "Today is Monday" in the system prompt. Pass the date in the final user turn instead, or round it to the day.
• A random conversation ID, request ID, or UUID prepended for logging.
• Dict or JSON serialization with unstable key order. Use sort_keys=True.
• Session state you append to as the conversation grows, placed before stable content.
• Tool schemas rebuilt dynamically from a database where column order varies between calls.

7. Application-Level Response Caching

Provider caching still makes a model call every time. If your workload has genuinely repeated requests — the same question asked by different users, a re-run of a batch job, a retry after a timeout — cache the response itself and skip the API entirely:

import hashlib, json, os
import redis

r = redis.Redis(host=os.getenv("REDIS_HOST", "localhost"), decode_responses=True)

def key(messages, model, temperature):
    payload = json.dumps({"m": messages, "model": model, "t": temperature},
                         sort_keys=True, ensure_ascii=False)
    return "llm:" + hashlib.sha256(payload.encode()).hexdigest()

def ask(messages, model="gpt-4o", temperature=0.0, ttl=3600):
    k = key(messages, model, temperature)
    if cached := r.get(k):
        return json.loads(cached)
    resp = client.chat.completions.create(
        model=model, messages=messages, temperature=temperature)
    out = resp.choices[0].message.content
    r.setex(k, ttl, json.dumps(out))
    return out

Two rules keep this safe. Only cache at temperature 0 — a nonzero temperature makes identical inputs legitimately produce different outputs, and caching one of them is a correctness bug. And include the model version in the key, so a provider rollout does not serve stale answers from the old model.

Use Redis over a local dict the moment you run more than one worker process. A per-process dictionary in a four-worker deployment gives you four separate caches, four times the misses, and a hit rate that looks worse in production than in your single-process testing.

8. Caching in Agents and RAG

Agents are the highest-value target because they resend the same prefix on every step of a loop. A ten-step agent run with a 4,000-token system prompt plus tool schemas would otherwise pay for 40,000 input tokens before the user's question is counted once. Keep the system prompt and tool definitions pinned at the front, append each new turn at the end, and never rewrite history mid-run — even a reworded assistant turn invalidates the rest of the prefix.

For RAG, sort retrieved chunks deterministically and cap the context window to a fixed size. A variable number of chunks changes the prefix length and the content, which breaks matching on nearly every query. If you want the deduplication benefits of semantic caching — matching "how do I reset my password" against "password reset steps" — treat it as a separate layer from prefix caching, and be aware it can return a cached answer to a question that merely sounds like the cached one. Prefix caching is exact and safe; semantic caching is approximate and needs a similarity threshold you tune against real data.

9. Common Mistakes

Mistake 1: Caching a Prefix That Changes Every Request

The classic. A timestamp in the system prompt gives you a cache write on every call and zero reads, which is more expensive than no caching at all. Read cache_read_input_tokens before assuming it works.

Mistake 2: Assuming OpenAI Caching Is Happening

It is automatic and silent, so a broken prefix just means full price forever with no error. Log prompt_tokens_details.cached_tokens and alert when the ratio drops.

Mistake 3: Caching Short Prompts

Under roughly 1,024 tokens there is no caching, only overhead. If your system prompt is 200 tokens, caching is not your cost problem — your cost problem is somewhere else.

Mistake 4: Interpolating User Data Into the System Prompt

Putting the username, tenant ID, or current date in the system block feels tidy and destroys the prefix. Keep it static; pass variables in the final user turn.

Mistake 5: Setting Too Many Cache Breakpoints

Each breakpoint is a separate cache entry with its own write premium. Four is the limit and two is usually right: one after the system block, one at the end of the previous turn.

Mistake 6: Caching Responses at Nonzero Temperature

Application-level response caching with temperature=0.7 freezes one sample out of many and presents it as the answer. Restrict it to temperature 0 or to batch jobs where you have explicitly accepted the trade-off.

Mistake 7: Ignoring Low-Traffic Endpoints

With a 5-minute TTL, an endpoint handling one request every ten minutes gets a write every time and never a read. Turn caching off for those paths rather than paying the premium for nothing.

Bottom Line

Prompt caching is the highest-return change available to most LLM applications, and it is mostly a prompt-architecture problem rather than a code problem. Put static content at the front, volatile content at the back, remove anything that varies from the prefix, and log your cached-token ratio so you can see when it breaks.

Do this in order: measure your current cached-token ratio (ten minutes), reorder your prompt so the static block is a true prefix (an hour), add cache breakpoints on Anthropic if that is your provider (twenty minutes), and only then add Redis response caching for genuinely repeated requests. Steps one through three are where nearly all of the savings are.