Semantic Caching for Agents

Introduction

L214 cached the repeated prefix — but you still called the model every time (just cheaper on the input). This lesson goes one level further: when a new query means the same thing as one you've already answered, return the stored answer and skip the model call entirely. That's semantic caching, and on the right workload it's the difference between a 500-2000ms, full-price LLM call and a 3-8ms, ~free cache lookup.

It reuses machinery you already know. Semantic caching is just RAG turned inward (Container 2): embed the query, vector-search a store, take the nearest neighbor by cosine similarity — but instead of retrieving documents, you retrieve a past question's answer. The new twist, and the danger, is the similarity threshold: "close enough" is a judgment call, and getting it wrong serves confidently wrong answers.

In this lesson:

The mechanism — embed → search → threshold → HIT (reuse answer) or MISS (call + store)
The three cache layers — KV (L208), prompt/prefix (L214), and semantic (the answer) — and how they stack
The payoff — skipping the call entirely, and realistic hit rates
The dial & the danger — the threshold, and false positives (embedding-close ≠ same intent)
For agents — where it helps, and why agent queries make it harder

Scope: this is answer caching. Prompt/prefix caching (reuse the computation, L214) is the sibling we just covered — we'll contrast them. Model routing (use a cheaper model, L216) is next. Embeddings, vector search, and cosine similarity come from Container 2 (RAG) — we use them here, not re-derive them.

Infographic titled 'Semantic Caching for Agents'. The big idea: when a new query MEANS the same thing as one you've already answered, return the stored ANSWER and skip the model call entirely. CENTER, the flow: a new query is embedded into a vector, that vector is searched against a store of past query embeddings, and the cosine similarity to the nearest one is compared to a THRESHOLD. If similarity is at or above the threshold it's a HIT — return the cached answer in about 3 to 8 milliseconds at roughly zero cost; if below, it's a MISS — call the LLM (500 to 2000 milliseconds, full cost) and store the new answer for next time. This is different from the previous lesson's prompt caching, and the contrast is the heart of the lesson: THREE CACHE LAYERS that stack — the KV cache (GPU memory, within one request, L208), the prompt/prefix cache (reuses the prefill of a shared prefix across requests, still calls the model, L214), and the semantic cache (application layer, reuses the whole ANSWER and skips the model, this lesson). THE DIAL is the similarity threshold: too low invites FALSE POSITIVES — the cache returns a confidently wrong answer because two queries are embedding-close but mean different things (the classic trap: 'weather in Paris, Texas' matches the cached 'weather in Paris, France' answer); too high collapses the hit rate. Start around 0.92 for factual workloads, monitor the false-positive rate, and adjust in 0.01 steps. THREE CARDS. Card 1, the payoff: a hit skips the call — about 100 percent cheaper and roughly 200 times faster; hit rates are about 50 to 70 percent for FAQ bots, 40 to 65 percent for agent tool calls, 30 to 50 percent for RAG over a static corpus. Card 2, the dial: the threshold trades savings for correctness; embedding-close is not the same as same-intent, and sometimes no threshold cleanly separates a valid paraphrase from a wrong-intent lookalike. Card 3, when NOT to use it: creative generation (every request is meant to differ), stateful multi-turn chat (each turn is unique), and personalized or time-sensitive answers (you risk serving the wrong user's or a stale answer) — and always add time-based eviction. FOR AGENTS: agent tool calls and repeated sub-queries cache well, but agent queries are contextual, so canonicalize the intent before matching and remember that changing context can change the right answer. Section roadmap strip: token economics, prompt caching, semantic caching, model routing, batch and distillation. Takeaway banner: semantic caching reuses the answer for a similar query and can skip a large fraction of calls — but tune the threshold, guard against false positives, evict stale entries, and never cache personalized or creative output.

The Idea — Reuse the Answer, Not Just the Prefix

Exact-match caching (a plain key-value store keyed on the literal prompt string) only helps when the exact same string repeats. But users ask the same thing in a hundred ways — "capital of France?", "what's France's capital", "France capital city". Exact match sees three different keys and three cache misses. Semantic caching catches all three because they embed to nearly the same vector.

The flow on every request:

Embed the query into a vector (any embedding model — the same kind you used for RAG).
Search your store of past query embeddings for the nearest neighbor (a vector DB / Redis).
Compare its cosine similarity to a threshold.
HIT (≥ threshold): return the cached answer — no model call. MISS (< threshold): call the LLM, then store (query embedding → answer) for next time.

Here's a minimal implementation — the whole technique is ~20 lines (production uses GPTCache, a Redis semantic cache, or an LLM gateway, but this is exactly what they do):

import numpy as np, anthropic

client = anthropic.Anthropic()
cache: list[tuple[np.ndarray, str, str]] = []   # (query_embedding, query, answer)
THRESHOLD = 0.92                                  # the dial — see "The Danger"

def embed(text: str) -> np.ndarray: ...           # your embedding model (Voyage, BGE, etc.)
def cosine(a, b): return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

def ask(query: str) -> str:
    qe = embed(query)
    # 1-3. find the most-similar PAST query
    best_sim, best_ans = max(((cosine(qe, e), a) for e, _, a in cache), default=(0.0, None))
    # 4a. HIT — reuse the stored answer, skip the model entirely (~free, ~5ms)
    if best_sim >= THRESHOLD:
        return best_ans
    # 4b. MISS — call the model, then index the answer by the QUERY embedding
    resp = client.messages.create(model="claude-opus-4-8", max_tokens=512,
                                  messages=[{"role": "user", "content": query}])
    answer = next(b.text for b in resp.content if b.type == "text")
    cache.append((qe, query, answer))
    return answer

Note what gets stored: the answer is indexed by the query's embedding (not the answer's). A future query that lands near it reuses the answer. That's the entire idea — and where you set THRESHOLD is the whole game.

The Payoff — Skip the Call Entirely (and the Three Cache Layers)

A semantic-cache hit doesn't make the model cheaper — it removes the model from the loop. The numbers are lopsided:

	Cache HIT	LLM call (MISS)
Latency	3-8 ms	500-2,000 ms
Cost	~free (an embed + a vector search)	full input + output

So a hit is ~100% cheaper and ~200× faster than generating. And because it caches the answer, it dodges the L213 problem that prompt caching couldn't — it saves the output tokens too (which often dominate the bill). Realistic hit rates depend on how repetitive the workload is:

FAQ / support bots: 50-70% (lots of repeated questions)
Agent tool calls: 40-65% (repeated sub-queries)
RAG over a static corpus: 30-50%

At a 60% hit rate you're not calling the model on 6 of every 10 requests — a direct, large cut to both the bill and p50 latency.

The three cache layers, finally side by side (they stack — production runs all three):

KV cache (GPU memory, L208) — reuse attention within one generation.

Prompt / prefix cache (inference layer, L214) — reuse the prefill of a shared prefix across calls (still calls the model, cheaper input).

Semantic cache (your application layer, this lesson) — reuse the whole answer for a similar query (skips the model).
Each one removes a different kind of recomputation: attention, then prefix, then the entire call.

The Dial — The Similarity Threshold

Everything hinges on one number: the cosine-similarity threshold at which you call two queries "the same." It's a direct savings-vs-correctness trade:

Threshold too low → you match queries that aren't really the same → false positives: the cache returns a confidently wrong answer. (This is the dangerous failure — covered next.)
Threshold too high → only near-identical queries match → the hit rate collapses and you've gained little.

There's no universal value, but a solid starting recipe (well-supported in practice):

Start at ~0.92 for any factual workload (where a wrong answer is costly).
Monitor the false-positive rate — sample cache hits and check the answer was actually right — for ~48 hours.
Adjust in 0.01 increments, watching hit rate and false positives together.
Adapt by task: code/factual answers want a higher threshold (accuracy matters); casual Q&A can run lower (more hits, low harm). Some systems set the threshold per query type.

A useful data point: tuning studies find a threshold around 0.8 can reach a ~69% hit rate at >97% accuracy on forgiving Q&A — but a factual product usually wants 0.92+ and accepts a lower hit rate to avoid wrong answers. You are choosing how much correctness to risk for savings. Tune it like an SLO, not a constant.

See It — The Semantic Cache Lab

Cached questions sit as points in embedding space. Send a new query and drag the threshold — watch it HIT (reuse the answer) or MISS (call the model), and watch the false positive appear:

Past Q→A pairs live as points in embedding space. Pick an incoming query — an exact repeat, a paraphrase, a different-intent lookalike ('weather in Paris, Texas'), or a novel one — and drag the similarity threshold. The query embeds, matches its nearest cached entry, and HITS (reuse the answer, ~free, 3–8ms) if similarity ≥ threshold, else MISSES (call the LLM). The trap: lower the threshold for more hits and the Paris-Texas query starts hitting the Paris-France answer — a false positive that serves a confidently wrong answer. Notice it's even embedding-CLOSER than the valid paraphrase, so no threshold cleanly separates them. Illustrative 2-D embedding.

The lab makes the danger unforgettable:

A paraphrase ("what's France's capital") sits close to the cached "capital of France?" → a correct HIT at a sensible threshold. That's the win.
But "weather in Paris, Texas" lands even closer to the cached "weather in Paris" (France) answer — so lowering the threshold to catch the paraphrase also serves the wrong city's weather. A false positive.
The kicker: in the lab the wrong-intent query is more embedding-similar than the valid paraphrase — so no single threshold cleanly separates them. That's the core truth: embedding-close ≠ same intent.

The Danger — Embedding-Close ≠ Same Intent

Semantic caching's failure mode is uniquely nasty: it doesn't error, it doesn't go blank — it returns a fluent, confident, wrong answer, and you won't notice without monitoring. Three ways it bites:

False positives (intent mismatch). Two queries can be lexically/embedding similar but semantically distinct — "Paris, Texas" vs "Paris, France"; "reset password" vs "reset 2FA". A too-low threshold serves the wrong cached answer.
Personalization. If the query encodes user-specific context, two users' prompts look nearly identical but want different answers — the cache can hand user B the answer it computed for user A. Never semantically cache personalized output (key by user, or don't cache it).
Staleness. A cached answer can go out of date ("latest iPhone?", today's price, current status). Add time-based eviction — e.g. 24h for news, 72h for semi-stable, 7d for static FAQs — and watch for semantic drift (if your traffic's meaning shifts, old entries mislead).

And some workloads you simply shouldn't semantically cache at all:

Creative generation (temperature > 0.5): every request is meant to differ → ~95% miss rate anyway.
Stateful multi-turn chat: each turn depends on the full history, so every query is unique by design.
Anything personalized or time-sensitive (per above).

The guardrails that make it safe: a conservative threshold (0.92+ for factual), time-based eviction, never cache personalized/creative output, and monitor the false-positive rate continuously. A semantic cache you don't monitor is a wrong-answer generator you're paying less for.

For Agents — Helpful, but Harder

Why does the lesson title say "for agents"? Because agents are both the biggest beneficiary and the trickiest case. From L213, agents are the cost bomb — long loops, many calls, lots of repeated sub-queries (the same tool call, the same "summarize this" on similar inputs). Semantic caching can short-circuit those repeats: agent tool-call hit rates run ~40-65%, directly cutting both the agent's cost and its step latency.

But agent queries make naive semantic caching fail more often, for a specific reason: an agent's query is contextual and stateful — the same surface text can mean different things depending on where the agent is in its task, what it just read, or which user it's serving. Embedding the raw query alone throws that context away, so you get more false positives. The fix the research points to is intent canonicalization: before matching, normalize the query to its structured intent (e.g. extract the actual operation + key arguments, drop irrelevant phrasing) and cache on that — so "get the weather in Paris, TX" and "Paris, France weather" canonicalize to different keys even though they embed close.

Practical rules for agents: cache idempotent, context-free sub-queries (a definition lookup, a stable tool result) aggressively; don't cache anything whose answer depends on the agent's mutable state or the specific user/session; and consider canonicalizing intent rather than embedding raw text. Same lesson as always — embedding-close is not same-intent, and agents have more context that the embedding doesn't see.

🧪 Try It Yourself

Reason through these, then confirm with the lab:

Predict: in the lab, set the Paraphrase query and lower the threshold until it HITs. Now switch to Different intent at that same threshold — what happens, and why is it impossible to fix by threshold alone?
How is semantic caching different from the prompt caching of L214 — what does each reuse, and which one saves output tokens?
A FAQ bot gets a 65% hit rate and saves a fortune. You add the same cache to a personalized dashboard assistant and users start seeing each other's data. What happened?
Your cache serves a stale answer to "what's our current on-call rotation?". What guardrail was missing?
Your agent's tool-call cache has a high hit rate but occasionally acts on the wrong cached result. What's the agent-specific fix?

→ (1) Lowering the threshold makes the paraphrase HIT (good) — but the Different-intent query ("Paris, Texas") is even closer to the cached Paris-France answer, so it HITs too and serves the wrong city's weather (false positive). No single threshold separates them because embedding similarity ≠ intent. (2) Prompt caching reuses the prefix computation (still calls the model; saves input only). Semantic caching reuses the whole answer (skips the model; saves input and output). (3) You semantically cached personalized output — different users' prompts embed nearly identically, so the cache returned one user's answer to another. Never cache personalized output (key by user, or skip caching). (4) Time-based eviction — a rotation changes; the entry should expire (e.g. hours), and you should watch for staleness. (5) Intent canonicalization — normalize the query to its structured intent (operation + args, and context/state) before matching, so context-different queries don't collide; and don't cache state-dependent results.

Mental-Model Corrections

"Semantic caching = prompt caching." No. Prompt caching reuses the prefix computation (still calls the model, L214). Semantic caching reuses the whole answer (skips the model). Different layers; they stack.
"A cache always saves money safely." A semantic cache can serve confidently wrong answers (false positives). Un-monitored, it's a wrong-answer generator you pay less for.
"Higher similarity = same question." Embedding-close ≠ same intent. "Paris, TX" ≈ "Paris, France" in vector space but means something different — sometimes more similar than a true paraphrase.
"Just turn it on everywhere." Don't cache personalized (wrong-user risk), time-sensitive (stale), creative (95% miss), or stateful multi-turn (unique by design) workloads.
"Set the threshold once." Tune it like an SLO: start ~0.92 for factual, monitor the false-positive rate, adjust in 0.01s, and consider per-task thresholds.
"It's a different technology from RAG." It's the same machinery (embed + vector search + cosine), pointed at past Q&A instead of documents.
"Agents are the easy case." They're the biggest win but the hardest — contextual/stateful queries cause more false positives; canonicalize intent, and don't cache state-dependent results.

Key Takeaways

Semantic caching reuses the answer for a similar query — embed → vector-search → threshold → HIT (return cached answer, 3-8ms, ~free) or MISS (call the LLM + store). It's RAG pointed at past Q&A.
A hit skips the model entirely — ~100% cheaper, ~200× faster, and it saves output tokens (unlike prompt caching). Hit rates: FAQ 50-70%, agents 40-65%, RAG 30-50%.
Three cache layers stack: KV (within-request, L208) → prompt/prefix (across-request prefix, L214) → semantic (the answer, here). Each removes a deeper level of recomputation.
The threshold is a savings-vs-correctness dial. Too low → false positives (wrong answers); too high → hit rate collapses. Start ~0.92 for factual, monitor false positives, tune in 0.01s.
The danger: embedding-close ≠ same intent. Guard with a conservative threshold, never cache personalized / time-sensitive / creative / stateful output, add time-based eviction, and monitor.
For agents: big win on repeated sub-queries, but contextual queries cause more false positives → canonicalize intent; don't cache state-dependent results.
Next — L216: Model Routing & Cascades — instead of caching the call, send each query to the cheapest model that can handle it — small models for easy work, escalate only when needed.