Adding Caches (Prompt + Semantic)

Introduction

You've built the gateway (L219); now we add the layer that lives inside it and, more than any other, decides whether your app is fast and cheap or slow and expensive: the cache. Chip Huyen calls it "perhaps the most underrated component of an AI platform."

You already know the two main caching mechanisms from the Cost section:

Prompt caching (L214) — reuse the model's prefill of a shared prefix (still calls the model, cheaper input).
Semantic caching (L215) — reuse the whole answer for a query that means the same thing (skips the model).

This lesson isn't about re-deriving those — it's about assembling them into a cache layer and operating it, which is a different, very real skill. The architectural questions are: which caches, in what order? what goes in the cache key? when does an entry expire? what should you never cache? Get these right and you get a 60–70% hit rate; get them wrong and you serve stale or wrong answers to users.

In this lesson:

Three caches, one layer — exact, prompt/prefix, and semantic, and the crucial split between them
The order of checks — how a request flows through the layer
The two hard parts — cache-key design and invalidation (the famous "two hard things in computer science")
What to cache and what never to — the personalization trap, architecturally
Hit rate — the number you design for and measure

Scope: the mechanisms are L214 (prompt) and L215 (semantic) — we'll recap in a line and build on them. Where the cache lives (the gateway) is L219. Watching hit rates is observability (L223). Here: how the caches compose and operate as a layer.

Infographic titled 'Adding Caches — Prompt + Semantic', the cache layer of a production LLM application. The big idea: compose THREE caches into one layer that sits in the gateway and skips or cheapens model calls. THE ORDER OF CHECKS, left to right: a request first hits the EXACT cache (a hash of the exact request; an identical repeat returns the stored answer in under a millisecond, for free); on a miss it falls through to the SEMANTIC cache (embed the query, vector-search past queries, and if cosine similarity is above a threshold return the cached answer — this catches PARAPHRASES that exact-match misses, like 'capital of France?' versus 'France's capital city'); on a miss there, it calls the MODEL — and even that call is made cheaper by the provider PROMPT cache, which reuses the prefill of a shared prefix (Anthropic about 90 percent off cached input, OpenAI about 50 percent off) while still calling the model. After the model answers, the result is STORED back into the exact and semantic caches so future repeats and paraphrases hit. The key distinction: the exact and semantic caches are APPLICATION-layer caches that SKIP the model entirely (about free, a few milliseconds); the prompt cache is a PROVIDER-layer cache that makes the call itself cheaper. THE TWO HARD PARTS, the operational reality: first, CACHE-KEY DESIGN — the key must hash EVERYTHING that changes the output: the user prompt, the SYSTEM-PROMPT hash, the model id, and the parameters; the classic bug is updating the system prompt without changing the key, so the cache keeps serving outdated answers. Second, INVALIDATION, TTL and EVICTION — give entries a time-to-live, but use TIERED TTLs by category (news a day, semi-stable a few days, static FAQs a week) because one TTL for everything is a top mistake; and evict with LRU when the cache fills. WHAT TO CACHE: high-reuse, factual, stable queries — yes; personalized, creative, fresh-real-time, or stateful queries — never, because semantic caching them returns plausible but contextually wrong answers. HIT RATE is the number you design for and measure (real systems reach roughly 60 to 70 percent), and the embedding model quality matters more than any other tuning choice; track it in observability. Roadmap strip: reference architecture (L218), gateway and router (L219), adding caches (L220), orchestration (L221), guardrails (L222), observability (L223). Takeaway banner: compose an exact cache, a semantic cache, and the provider prompt cache into one layer in your gateway — exact catches repeats, semantic catches paraphrases, prompt cache cheapens the misses — but the real work is cache-key design and invalidation, and never caching personalized or fast-changing answers.

Three Caches, One Layer

"Caching" in an LLM app isn't one thing — it's three different caches that catch different things and live at different layers. The single most important distinction:

Cache	What it stores	What it catches	Effect	Layer
Exact	the answer, keyed by a hash of the exact request	identical repeats	skips the model (<1ms, free)	application
Semantic	the answer, keyed by the query embedding	paraphrases (same meaning, different words)	skips the model (~8ms, free)	application
Prompt / prefix	the model's prefill of a shared prefix	a repeated prefix (system prompt, long doc)	cheapens the call (still runs)	provider

Read the Effect column twice — it's the whole lesson:

The exact and semantic caches are application-layer caches that skip the model entirely — a hit is ~free and milliseconds fast.
The prompt cache is a provider-layer cache that doesn't skip the model — it makes the call you do make cheaper (Anthropic ~90% off cached input, OpenAI ~50%).

So they're complementary, not competing — you run all three. Exact + semantic try to avoid the call; when you can't avoid it, prompt caching makes it cost less. The new one here is the exact cache — the simplest and safest: a plain key-value store (Redis, in-memory) keyed by a hash of the request. Zero risk of a wrong answer (the request was identical), sub-millisecond, and it should usually be your first check.

The Order of Checks

Composed, the cache layer is a waterfall: try the cheapest, safest cache first; fall through to the next; only call the model if everything misses — then store the answer so the next repeat or paraphrase hits.

def answer(query: str, ctx: Context) -> str:
    key = cache_key(query, ctx)              # ← the hard part — see "Cache-Key Design"

    # 1. EXACT cache — identical request? return instantly (sub-ms, free, zero risk)
    if hit := exact_cache.get(key):
        return hit

    # 2. SEMANTIC cache — a PARAPHRASE of something we've answered? (embed + vector search)
    qe = embed(query)
    match, sim = semantic_cache.nearest(qe)
    if sim >= THRESHOLD:                      # tune to avoid false positives (L215)
        return match.answer

    # 3. MISS — call the model. The provider PROMPT cache (L214) makes THIS call cheaper.
    out = gateway.complete(query, ctx)        # prompt-cache the shared prefix

    # 4. STORE the answer in BOTH app caches so future repeats/paraphrases hit
    exact_cache.set(key, out, ttl=ttl_for(ctx))      # ← tiered TTL — see "Invalidation"
    semantic_cache.add(qe, out, ttl=ttl_for(ctx))
    return out

Why exact before semantic? Because exact is faster (a hash lookup vs an embedding + vector search) and risk-free (the request was identical), so it should short-circuit the common case before you pay for the smarter, fuzzier check. And why store into both? So the next identical request is an exact hit and the next reworded one is a semantic hit. This waterfall is exactly what the Cache Layer Lab below plays out — watch each request land in one of the three buckets.

Hard Part #1 — Cache-Key Design

Here's a deceptively deep question: what is two requests being "the same"? Your cache key is the answer, and getting it wrong is the #1 caching bug. The rule:

The cache key must hash everything that can change the model's output. If a factor affects the answer but isn't in the key, the cache will serve an answer computed under different conditions — silently wrong.

What "everything" usually includes:

the user prompt (obviously), and the retrieved context if it's part of the prompt;
the system prompt — or better, a hash of it (this is the classic trap below);
the model id (gpt-5.5 and claude-opus-4-8 give different answers);
the parameters that change output — temperature, max_tokens, tools, response format;
the user / tenant id if answers are personalized (see "What to Cache").

import hashlib, json

# ✗ THE CLASSIC BUG: key = just the user's text.
#   You update the SYSTEM PROMPT ("now answer in French") and deploy — but the key
#   didn't change, so the cache keeps serving the OLD English answers for hours.
bad_key = hashlib.sha256(query.encode()).hexdigest()

# ✓ CORRECT: hash every input that can change the output.
def cache_key(query: str, ctx: Context) -> str:
    material = json.dumps({
        "q": query,
        "system": ctx.system_prompt,      # include it — or a version tag/hash of it
        "model": ctx.model,               # different model → different answer
        "temperature": ctx.temperature,   # and any param that changes output
        "tools": ctx.tool_names,
        "user": ctx.user_id if ctx.personalized else None,   # per-user only when needed
    }, sort_keys=True)
    return hashlib.sha256(material.encode()).hexdigest()

A useful shortcut for the system prompt: bump a prompt_version tag whenever you change it — incrementing the version changes every key, instantly invalidating the whole cache for that prompt. That turns "I changed the prompt and now the cache is poisoned" into a one-line config change.

Hard Part #2 — Invalidation, TTL & Eviction

"There are only two hard things in computer science: cache invalidation and naming things." For an LLM cache, invalidation is about staleness — a cached answer can quietly go out of date. Three tools manage it:

TTL (time-to-live) — every entry expires after some lifespan. The tension: too long → stale answers; too short → low hit rate. And the #2 caching mistake (after bad keys) is one TTL for everything. Different data goes stale at different rates, so use tiered TTLs by category:
- Real-time (prices, weather, status): minutes — or don't cache.
- Semi-stable (docs, policies): hours to days.
- Static (definitions, FAQs, historical facts): days to weeks.
Active invalidation — when the underlying data changes (a doc is edited, a product is updated), evict the affected entries rather than waiting for TTL. (This needs a link from your data to its cache entries — tag entries by source.)
Eviction (size, not freshness) — when the cache is full, something must go. LRU (least-recently-used) is the default; LFU and FIFO are alternatives. Pick by your access pattern.

The honest take: a cache is a bet that the world hasn't changed since you stored the answer. TTL and invalidation are how you bound that bet. The mature move is to tag every entry with a category and a source, give each category its own TTL, and evict on both time (TTL) and events (source changed) — not one global timer.

What to Cache — and What Never To

A cache only helps when the same answer is correct for repeated (or similar) requests. That's true for a lot of traffic — and dangerously false for some. Architecturally, gate caching on the type of query:

Cache aggressively:

Factual / stable lookups (definitions, docs, FAQs, "what does this error mean").
Repeated sub-tasks (the same classification, the same "summarize this standard doc").
Expensive, deterministic steps (a tool result, a SQL query's narrative, an embedding).

Never cache (or you'll serve confidently wrong answers):

Personalized output — two users' prompts can look identical but want different answers. A shared semantic cache will hand user B the answer it made for user A. (Either key by user_id, or don't cache.)
Real-time / time-sensitive — "current price," "today's status." Stale by definition; short TTL or skip.
Creative generation (temperature high) — every request is meant to differ; you'd get a ~95% miss rate anyway and kill the variety.
Stateful multi-turn — each turn depends on the full conversation, so "the same" question isn't the same.

The trap that ties these together is the one from L215: "looks the same" ≠ "same correct answer." The exact cache is safe because it requires an identical request; the semantic cache is where this bites — a loose threshold on personalized or time-sensitive data is how you ship a plausible but wrong answer. When in doubt, don't cache it — a cache miss costs money; a false cache hit costs trust.

See It — The Cache Layer Lab

Play a realistic stream of queries — exact repeats, paraphrases, and novel ones — through the composed layer and watch each one land as an exact hit, a semantic hit, or a miss, while the hit rate climbs:

Play a realistic stream of queries through the composed cache layer. Each request checks the **exact cache** (identical repeat → instant, free), then the **semantic cache** (a paraphrase of something already answered → free), else **MISS** → call the model and store the answer. Watch the **hit rate climb** as the cache fills, and watch *which* cache catches *what*: **exact** handles literal repeats, **semantic** catches the reworded ones exact-match would miss. That's the whole point of composing them — they catch **different** things, so together they reach the ~60–70% hit rate real systems see. (Illustrative costs/latencies.)

Two things to notice. (1) The cache fills. The first time you see any query it misses — so an empty cache is worthless; hit rate is a property of warm traffic. (2) The two caches catch different things. Pure exact-match would miss every reworded query ("capital of France?" vs "France's capital city"); the semantic cache catches those. That's why you compose them — and why the combined hit rate (~60–70% here) beats either one alone.

Hit Rate Is a Number You Design For

A cache layer isn't "on or off" — it has a hit rate, and that number is a first-class metric you tune and monitor. It's also the only thing that determines your payoff:

$\text{cost saved} \approx \text{hit rate} \times \text{calls} \times \text{cost/call}$

At a 60% hit rate, you're not calling the model on 6 of every 10 requests — a direct 60% cut to that slice of the bill and a huge p50 latency win (a hit is ~200× faster than a generation). Realistic hit rates: FAQ/support 50–70%, agent sub-tasks 40–65%, RAG over a static corpus 30–50%.

What actually moves the number:

Embedding quality (semantic cache) — "the embedding model determines cache quality more than any other tuning choice." A better embedding = more true paraphrases caught at a safe threshold.
The threshold (semantic) — lower = more hits and more false positives; tune it like an SLO (L215).
TTL & key granularity — too-tight keys or too-short TTLs quietly starve the hit rate.

Measure it or you're flying blind. Track hit rate, false-positive rate, and stale-serve rate per cache, broken down by query type — this is a core piece of the observability layer (L223). A cache you don't monitor will silently drift from "60% savings" to "serving last week's answers," and you won't know which.

🧪 Try It Yourself

Reason through these, then confirm with the Cache Layer Lab:

Predict: you deploy with a brand-new (empty) cache. What's the hit rate on the first 100 unique users, and why does that not mean caching is broken?
Why run an exact cache and a semantic cache — wouldn't semantic alone catch everything exact does?
You change your system prompt from "be concise" to "answer in French." Users keep getting English answers for an hour. What went wrong, and what's the one-line fix?
Your FAQ cache is great, but your "what's my account balance?" answers start showing other users' balances. What happened?
A teammate sets a single 24-hour TTL for all cached answers. Why is that a problem, and what's the better pattern?

→ (1) Near 0% — every query is the first of its kind, so all misses. Caching pays off on warm, repeated traffic, not cold starts; hit rate climbs as the cache fills. (2) They catch different things and have different costs: exact is a sub-ms, zero-risk hash for identical repeats; semantic is a slower, fuzzier embedding search for paraphrases. Run exact first (fast, safe), semantic as the fall-through. (3) The cache key didn't include the system prompt (or a version of it), so it kept serving pre-change answers. Fix: add a prompt_version to the key and bump it (or hash the system prompt). (4) You cached personalized output in a shared cache — different users' prompts collided and one got another's answer. Key by user_id, or don't cache personalized responses. (5) Different data goes stale at different rates — a 24h TTL is too long for prices and too short for static FAQs. Use tiered TTLs by category (minutes / hours-days / days-weeks), plus event-based invalidation when the source changes.

Mental-Model Corrections

"Caching is one thing." It's three — exact (hash, skips model), semantic (embedding, skips model), prompt/prefix (provider, cheapens the model call). You run all three; they're complementary.
"Prompt cache and semantic cache are interchangeable." No — prompt cache still calls the model (cheaper input); semantic cache skips it. Different layers, different savings.
"Just cache the prompt text." The cache key must hash everything that changes the output — system prompt, model, params, (and user if personalized). The stale-system-prompt bug is the classic miss.
"Set a TTL and forget it." One global TTL is a top mistake. Use tiered TTLs by category plus event-based invalidation; evict on size with LRU.
"A cache always helps." Only when the same answer stays correct. Never cache personalized / real-time / creative / stateful output — a false hit serves a confidently wrong answer.
"Caching is on or off." It's a hit rate you design for and monitor (60–70% is good). Embedding quality and threshold move it most; track it in observability (L223).
"An empty cache should still help." A cold cache hits nothing — caching is a property of warm, repeated traffic.

Key Takeaways

The cache layer composes three caches in your gateway: exact (hash → skip model, <1ms, zero-risk), semantic (embedding → skip model, catches paraphrases, tune the threshold), and the provider prompt cache (reuse the prefix → cheapen the call). Exact + semantic avoid the call; prompt cache makes the unavoidable call cheaper.
Order of checks: exact → semantic → model (with prompt caching) → store the answer in both so future repeats and paraphrases hit. Exact first: faster and risk-free.
Hard part #1 — cache-key design: hash everything that changes the output (prompt, system-prompt version, model, params, user-if-personalized). The classic bug is a stale system prompt — fix with a prompt_version tag.
Hard part #2 — invalidation: tiered TTLs by category (not one global timer), event-based eviction when the source changes, and LRU when the cache fills. A cache is a bet the world hasn't changed.
Never cache personalized / real-time / creative / stateful output — "looks the same" ≠ "same correct answer." When in doubt, don't: a miss costs money, a false hit costs trust.
Hit rate is the metric — 60–70% is good; it determines your savings, climbs as the cache warms, and is moved most by embedding quality and threshold. Monitor it (L223).
Next — L221: Orchestration & Pipelines. We've got the front door, the router, and the cache; now the glue that chains all the components into one end-to-end pipeline.