Prompt Caching

Introduction

L213 ended on the budget-wrecker: the API is stateless, so you re-send and re-pay for the entire prompt on every single call — the system prompt, the tool definitions, the RAG context, the conversation history. Prompt caching is the direct fix, and the highest-ROI cost lever there is: cache that repeated prefix once and pay ~0.1× (a 90% discount) to reuse it, instead of full price every time.

If this sounds familiar, it should. Prompt caching is the cross-request version of the KV cache from L208. Within one request, the KV cache stops the model re-reading the prompt every token; across requests, prompt caching stops you re-paying for the prefill of the same prefix every call. Same idea — don't recompute what hasn't changed — applied one level up.

In this lesson:

What it caches — the repeated prefix (system + tools + context), and why it's the L208 KV cache, persisted
The economics — read 0.1×, write 1.25-2×, break-even ~2 reuses — and the honest total-bill reality
How to use it — Anthropic's explicit cache_control, plus OpenAI/Gemini/DeepSeek
The golden rule — stable content first, volatile last
The #1 bug — silent cache misses that quietly take your hit rate to 0% (and make it worse than no cache)

Scope: this is prompt (prefix) caching — reusing the computation of a repeated prompt prefix. Semantic caching — reusing the answer for a semantically similar question (skipping the call entirely) — is a different technique, and it's L215 (next). We'll keep them straight.

Infographic titled 'Prompt Caching'. The big idea: stop re-paying full price for the same prompt prefix on every call — cache it once and read it back at a tenth of the cost. CENTER, a request drawn as a stack in render order: tools (definitions), system prompt, and RAG context together form a big STABLE prefix of about 8,000 tokens, then a small user question (about 100 tokens) that VARIES each call. A 'cache_control' breakpoint line sits right after the stable prefix. WITHOUT caching, all 8,000 prefix tokens are re-sent and re-billed at full price on every call. WITH caching, the first call WRITES the prefix to cache at 1.25x the input price, and every later call READS it at just 0.1x (a 90 percent discount) — so over many calls sharing the prefix the cost collapses (about 70 percent cheaper in the example, around 90 percent off the prefix itself). This is the cross-request version of the KV cache from the inference section: it reuses the prefill of the repeated prefix. THE HONEST REALITY: the 90 percent is only on the cached INPUT — output tokens are never cached and still dominate the bill, and the write costs extra, so the real total-bill saving is more like 30 to 70 percent; break-even is about 2 reuses (one call alone is a loss because of the write fee). THE NUMBER ONE BUG: a silent cache miss. If anything in the prefix changes every call — a timestamp or UUID in the system prompt, non-deterministically ordered tool definitions, or a sliding window that drops the oldest message — the byte prefix differs each time, so the cache hit rate is ZERO and you pay the write fee every call, which is MORE expensive than no caching at all. THREE CARDS. Card 1, the economics: read 0.1x, write 1.25x (5-min) or 2x (1-hour); only input is discounted. Card 2, the golden rule: put STABLE content first (tools, system, context), keep it byte-identical, and move anything that changes after the breakpoint; verify with usage cache_read_input_tokens. Card 3, the providers: Anthropic and Gemini are explicit (you mark what to cache), OpenAI and DeepSeek are automatic (no code). Section roadmap strip: token economics, prompt caching, semantic caching, model routing, batch and distillation. Takeaway banner: cache the stable prefix, keep it byte-identical, move variable content after the breakpoint, and verify the read — it is the highest-ROI cost lever, but only if it actually hits.

The Idea — Cache the Repeated Prefix

Most production prompts are mostly the same every call. A RAG assistant sends a 2,000-token system prompt + a 6,000-token retrieved context + the user's 100-token question. Across a session, the first 8,000 tokens barely change — only the question at the end does. Yet without caching, you pay full input price for all 8,000 every call.

Prompt caching exploits that. On the first call, the provider stores the processed prefix (technically, its KV cache — L208 — kept warm on the server). On subsequent calls with the same prefix, it skips the prefill and bills those tokens at ~0.1×. What's cacheable is exactly the stable, repeated stuff:

the system prompt (instructions, persona, rules),
the tool definitions (a big tool catalog can be 10k+ tokens — real money),
few-shot examples,
RAG / retrieved context that's reused across turns,
the fixed conversation prefix (early turns that don't change).

The catch that drives everything about how you use it: caching is a prefix match. The provider caches a contiguous span from the very start of the request. If a single byte changes anywhere in that span, the match breaks from that point on. So what you cache and where you put it is the whole skill — which is why the rest of this lesson is about placement and not breaking the match.

The Economics — Write Once, Read Cheap (Honestly)

Caching isn't free — there's a small write cost to store the prefix, then cheap reads. The rates (Anthropic; OpenAI/DeepSeek are similar but with no write fee):

Operation	Multiplier vs normal input	When
Cache write	1.25× (5-min TTL) / 2× (1-hour TTL)	the first call (storing the prefix)
Cache read	0.1× (90% off)	every later call that hits the cache
(uncached input)	1×	the varying part (the question)

So one call alone is a loss (you paid the 1.25× write and never reused it). Break-even is ~2 reuses; after that it's pure profit, and it compounds — 30 calls sharing an 8k prefix is ~86% cheaper on the prefix.

But here's the honest part most articles skip. The "90% savings!" headline is on the cached input tokens only. Two things bound your actual bill reduction:

Output is never cached (no provider caches output) — and from L213, output often dominates the bill.
Writes cost extra, and you only hit the cache on repeated prefixes.

The reality: a workload at an 80% hit rate where output dominates might see the "90% cached read" translate to only a ~30% total bill reduction; an input-heavy RAG workload (huge context, short answer) can see 60-70%+. Prompt caching is the best cost lever — but its impact is set by your input/output split (L213). It's huge when you're input-dominated, modest when you're output-dominated.

def cache_cost(prefix_tok, calls, write_mult=1.25, read_mult=0.1, in_rate=3):
    """Cost of a repeated prefix, uncached vs cached. rates in $ per 1M tokens."""
    full = prefix_tok / 1e6 * in_rate
    uncached = full * calls
    cached = full * write_mult + full * read_mult * (calls - 1)   # 1 write + (calls-1) reads
    saved = round((uncached - cached) / uncached * 100)
    return uncached, cached, saved

for calls in (1, 2, 5, 30):
    u, c, pct = cache_cost(8000, calls)   # an 8k-token stable prefix
    print(f"{calls:>2} calls:  uncached ${u:.3f}   cached ${c:.3f}   -> {pct:>3}% saved on the prefix")
# 1 call is a LOSS (you paid the write); break-even is ~2 reuses; it compounds after that.

How to Use It — Explicit (Anthropic) vs Automatic (OpenAI)

Providers split into two camps. Anthropic & Gemini are explicit — you mark what to cache. OpenAI & DeepSeek are automatic — they cache repeated prefixes for you, no code. Explicit gives you control (and the 1-hour TTL); automatic is zero-effort. Here's the explicit form on Claude — cache the stable prefix with a cache_control breakpoint, and verify it actually hit via usage:

import anthropic

client = anthropic.Anthropic()
SYSTEM = "...long, stable system prompt + rules..."        # ~2k tokens, same every call
DOCS   = "...retrieved RAG context for this session..."     # ~6k tokens, same every call

resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    system=[
        {"type": "text", "text": SYSTEM},
        # cache EVERYTHING up to & including this block (the stable 8k-token prefix):
        {"type": "text", "text": DOCS, "cache_control": {"type": "ephemeral"}},   # 5-min TTL
        # (use {"type": "ephemeral", "ttl": "1h"} for a 1-hour cache at a 2x write)
    ],
    messages=[{"role": "user", "content": "the question — VARIES, stays after the breakpoint"}],
)

# ALWAYS verify the cache hit — these three numbers tell the whole story:
u = resp.usage
print("cache_creation_input_tokens:", u.cache_creation_input_tokens)  # 1st call: ~8000 (billed ~1.25x)
print("cache_read_input_tokens:    ", u.cache_read_input_tokens)       # later calls: ~8000 (billed ~0.1x)
print("input_tokens (uncached):    ", u.input_tokens)                  # just the question (full price)
# If cache_read_input_tokens stays 0 across repeated calls -> a silent invalidator (see below).

OpenAI/DeepSeek need none of this — send the same prefix and they cache it automatically (you just see it in the usage breakdown). The trade: with explicit caching you choose the breakpoint and TTL; with automatic you take what you get. Either way, the discipline is the same — and it's the next section.

The Golden Rule — Stable First, Volatile Last

Because caching is a prefix match, there's one rule that everything follows from:

Put the stable content at the front, and anything that changes at the back.

Requests render in a fixed order — tools → system → messages — so structure them stable-to-volatile:

Front (cache this): tool definitions → system prompt → few-shot → reused RAG context. Frozen, byte-identical every call.
Back (don't cache): the user's current question, per-request IDs, timestamps, anything dynamic.

Put the breakpoint at the end of the stable span, and keep that span byte-for-byte identical. Two practical rules that fall out:

Never interpolate a dynamic value into the cached prefix — no f"Current time: {now}" in the system prompt. That single change re-breaks the cache every call.
Serialize tools/JSON deterministically (sort keys) so the bytes are identical across calls.

And then verify: read usage.cache_read_input_tokens on the 2nd+ call. If it's a healthy fraction of your prefix, you're caching; if it's 0, you have a silent miss — which is so common it gets its own section.

See It — The Prompt Cache Lab

Watch the economics — and the failure mode — directly. The request is a big stable prefix + a varying question. Switch modes and drag the number of calls:

A request is a stack: tools + system + RAG context form a big stable prefix, then the user's question varies. Pick a mode — No caching (pay full for the prefix every call), Cache the prefix (write once at 1.25×, then read at 0.1× → ~70% cheaper over many calls), or Cache + a timestamp in the prompt (the prefix changes every call → every call is a MISS that re-writes → ~20% MORE expensive than no cache). Drag the number of calls and watch cumulative cost, % saved, and cache hit rate, with a per-call strip showing write (1.25×) / read (0.1×) / full (1×). Illustrative, Sonnet pricing.

Three modes, three outcomes:

No caching — the baseline: full price for the 8k prefix on every call.
Cache the prefix — call 1 writes (amber, 1.25×), every call after reads (green, 0.1×) → ~70% cheaper over 30 calls, 97% hit rate. The more calls share the prefix, the bigger the win.
Cache + timestamp — the prefix changes every call, so it's a MISS every time (0% hit): you pay the write fee on every call and it ends up ~20% MORE expensive than no caching. That's the #1 bug — next.

The #1 Bug — Silent Cache Misses

Prompt caching fails silently. Your code looks right, the API returns 200, the answer is fine — but cache_read_input_tokens is 0 and your bill went up (you're paying write fees with no reads). It's always the same cause: something in the prefix changes every call, so the byte-match breaks. The three patterns that get everyone:

1. A mutable system prompt. A timestamp, UUID, request ID, or user ID interpolated into the cached prefix. Every call has a different prefix → hit rate 0%. Fix: move dynamic values to the user message (after the breakpoint); keep the system prompt static.

2. Non-deterministic tool serialization. Tool definitions (or any JSON in the prefix) rendered in a different order across calls — from dict ordering, a different library version, etc. Hit rate drops to 0-40%. Fix: sort deterministically (force key sort) so the serialized bytes are identical.

3. A sliding-window conversation history. You cap history by dropping the oldest message — but that shifts the whole prefix, so every cached byte after it is now misaligned. Fix: cache only the system + tools (simplest), or keep a fixed prefix + sliding tail (the first N messages stay put), or summarize dropped messages into the cached system block on refresh.

The debugging loop is dead simple and you should bake it into dev: make two identical calls and assert cache_read_input_tokens > 0 on the second. If it's 0, diff the rendered prefix bytes between the two requests — the difference is your invalidator. A cache you think is working but isn't is worse than no cache: you pay the write premium for nothing.

🧪 Try It Yourself

Reason through these, then confirm with the lab:

Predict: in the lab, Cache the prefix at 1 call vs 30 calls — when is caching a loss, and why?
Your system prompt starts with "Today is {date}. ". You enable caching and the bill rises. What happened, and what's the one-line fix?
You cache a 10k-token tool catalog. It hits in testing but only ~40% in production. What's the likely cause?
A RAG app (15k-token context, 150-token answer) vs a creative-writing app (200-token prompt, 2k-token answer): which gets more total-bill benefit from prompt caching, and why?
How do you prove your cache is working in code?

→ (1) At 1 call it's a loss — you pay the 1.25× write and never read it back; break-even is ~2 reuses. At 30 calls it's ~70% cheaper. (2) The date in the prefix changes every call, so it's a cache miss every time (0% hit) and you eat the write fee → more expensive. Fix: move the date out of the system prompt into the user message (after the breakpoint). (3) Non-deterministic tool serialization — the catalog renders in a different order across calls. Fix: sort keys so the bytes are identical. (4) The RAG app — it's input-dominated (huge cached prefix, tiny output), so the 90%-off-input discount hits most of the bill. The creative app is output-dominated (output isn't cached), so caching barely helps the total. (5) Make two identical calls and assert cache_read_input_tokens > 0 on the second; if 0, diff the prefix bytes to find the invalidator.

Mental-Model Corrections

"Prompt caching = caching the model's answers." No — that's semantic caching (L215). Prompt caching reuses the prefill of a repeated prompt prefix (the cross-request KV cache, L208); the model still runs and generates fresh output.
"It saves 90% on my bill." 90% is on the cached input tokens only. Output is never cached and often dominates (L213), and writes cost extra — so the total saving is typically 30-70%, set by your input/output split.
"Just turn caching on." A cache only helps if the prefix is byte-identical and reused. A single dynamic value (timestamp/ID) in the prefix → 0% hit rate → worse than off.
"One cached call saves money." No — one call is a loss (you paid the write). You need ≥2 reuses to break even.
"Where I put things doesn't matter." It's the whole game. Caching is a prefix match — stable content must come first, volatile content last.
"It worked in testing, so it's working." Verify in production with cache_read_input_tokens. Silent misses (tool ordering, sliding history) show up only at scale.
"Caching helps output too." No provider caches output. Caching is an input-side lever only.

Key Takeaways

Prompt caching reuses a repeated prompt prefix — the cross-request KV cache (L208). It's the direct fix for L213's "you pay the prompt every call," and the highest-ROI cost lever.
Economics: read 0.1× (90% off), write 1.25× / 2× (5-min / 1-hour TTL); break-even ~2 reuses. Only input is discounted — never output.
The honest reality: the 90% is on cached input; total-bill saving is 30-70%, set by your input/output split (L213). Huge for input-heavy RAG; modest when output dominates.
Golden rule: stable first, volatile last. Cache tools → system → context; keep it byte-identical; put the question/IDs/timestamps after the breakpoint.
The #1 bug is a silent miss: a timestamp/ID in the prefix, non-deterministic tool order, or a sliding-window history → 0% hit rate → worse than no cache. Verify cache_read_input_tokens > 0; diff the prefix to find the invalidator.
Providers: Anthropic/Gemini explicit (cache_control); OpenAI/DeepSeek automatic.
Next — L215: Semantic Caching for Agents — the other cache: when two questions mean the same thing, reuse the answer and skip the model call entirely.