Memory Stores & Retrieval (Vector + Summary)

Introduction

You now know memory comes in tiers (L133) and kinds (L134). This lesson is the machinery underneath: how memory is actually stored, and how the right piece is retrieved into the context window at the right moment.

Remember the constraint that started this whole section: the model can only see the window, and the long-term store can hold thousands of memories. So there's an unavoidable bottleneck — which handful of memories do you fetch? Get it wrong and the agent is either flooded with noise or missing the one fact that mattered.

Memory retrieval is RAG pointed at the agent's own past — with one crucial twist the document case never needed: time and importance matter, not just similarity.

Good news: you already learned most of the retrieval toolkit in the RAG section (Container 2) — embeddings, vector search, hybrid BM25 + dense, reranking, RRF/MMR. This lesson reuses all of it and adds what's special about memory.

In this lesson:

Where memory lives — the storage menu: vector, summary, structured/graph, and skill stores
Why retrieval is more than similarity — the relevance + recency + importance score (hands-on)
The full retrieval pipeline and the write path (consolidation, dedup, metadata)
2026 frameworks — Mem0, Zep, Letta — and the traps

Scope: this is how memory is stored and retrieved. We reuse the RAG section's embedding/search/rerank mechanics rather than re-derive them. The full hands-on build is next (L136 — Hands-On: A Memory-Aware Agent).

An infographic titled 'Memory Stores & Retrieval (Vector + Summary)' showing how an agent stores and fetches memory. On the left, a MEMORY STORE holds thousands of memories in different representations: a VECTOR store of dense embeddings for similarity search, SUMMARIES of compressed text, and a STRUCTURED or graph store for relationships and skills. Because the model can only see the context window, retrieval must pick the few memories that matter. The center shows the RETRIEVAL SCORE, the agent-memory insight that retrieval is more than similarity: a memory's score combines RELEVANCE (cosine similarity between the query embedding and the memory), RECENCY (an exponential-decay function so fresh memories rank higher), and IMPORTANCE (a self-assessed score from 1 to 10 that lifts core memories over mundane ones). This three-signal score comes from the Generative Agents paper (Park 2023) and is a substantial improvement over pure cosine similarity. The full retrieval pipeline runs left to right: formulate a good query (often an LLM rewrites the raw input), search with dense embeddings plus sparse BM25 and metadata filters on user, type, and time, score each candidate by relevance plus recency plus importance, rerank the shortlist with a cross-encoder, and inject only the top-k into the context window on the right. The write path matters too: filter low-signal records, canonicalize and deduplicate, tag with metadata, and periodically consolidate episodic logs into semantic facts. The three cards summarize the storage menu: a vector store scales to millions and answers what is most similar but loses structured relationships; summary memory is transparent and compact but suffers summarization drift as each pass discards detail; structured and graph stores answer what caused what and hold skills, at the cost of schema design. In 2026 the production stacks are hybrids: Mem0 blends vector and graph, Zep's Graphiti combines BM25, embeddings, and graph traversal with no LLM call at retrieval time, and Letta keeps core memory in the prompt with archival memory in a database. Takeaway: store memory as vectors, summaries, and structure, and retrieve it by relevance plus recency plus importance, reranked, so the right memory reaches the window at the right moment.

The Retrieval Problem

Start with the bottleneck. Your agent has accumulated a year of interactions — tens of thousands of episodic events, hundreds of semantic facts, a library of skills. The context window holds maybe a few dozen of them. Every turn, something must choose the few memories worth loading.

This is exactly the retrieval problem from RAG — except the corpus isn't a pile of documents, it's the agent's own memory, and that changes two things:

The corpus is dynamic and self-written. The agent generates its own memories as it runs (and must decide what to keep — the write path, later). A document corpus is static; a memory store is alive.
Recency and importance matter, not just relevance. For a document, only similarity to the query matters. For a memory, a fresh observation or a critical fact can outweigh a merely-similar one — "the user said they're exhausted yesterday" beats a vaguely-related note from six months ago.

So memory retrieval = the RAG pipeline + a memory-aware score. We'll take the storage first, then the scoring.

Where Memory Lives — The Storage Menu

There's no single "memory database." There are four representations, each answering a different question well — and real systems combine them:

Store	Good at	Weak at	Holds (kind)
Vector (embeddings + ANN)	"what's most similar?" — fuzzy semantic recall, scales to millions	"what caused what?" — loses structured relations	episodic events, facts
Summary (compressed text)	compact, transparent, zero infra	drift — each compression drops detail	rolling conversation, gist
Structured / graph (SQL, KV, KG)	precise queries, relationships ("failures of service X last week")	needs schema; not fuzzy	profiles, entities, links
Skill library (code)	reusable behavior, executed deterministically	only for procedures	procedural skills (Voyager)

The hybrid is standard. MemGPT layers a context window ("main memory") + a searchable recall DB + a vector archive; Mem0 blends a vector store with a knowledge graph; Letta keeps editable "core memory" in the prompt and the rest in a database. The title of this lesson — Vector + Summary — names the two you'll reach for most, so we'll go deep on those.

Vector Memory — Store as Embeddings

The workhorse. Vector memory is RAG's machinery applied to the agent's experience: embed each memory into a dense vector, store it in a vector index, and at retrieval time embed the query and pull the nearest neighbours by cosine similarity. (All the embedding + ANN details are from the RAG section — nothing new there.) The memory-specific parts are the metadata and the over-fetch:

# STORE a memory: embed it, attach metadata, write to the vector index.
store.add(
    text="User is vegetarian and allergic to peanuts.",
    embedding=embed(text),                        # the SAME embeddings you met in RAG
    metadata={"user_id": "u_42", "type": "semantic", "ts": now(), "importance": 9},
)

# RETRIEVE for this turn: embed the query, filter by metadata, over-fetch candidates.
hits = store.search(
    embedding=embed(user_msg),                    # cosine similarity (ANN)
    filter={"user_id": "u_42"},                   # scope to THIS user
    k=20,                                          # over-fetch, then re-score + rerank…
)

Two things to notice. Metadata is essential for memory (more than for plain doc-RAG): every memory carries user_id, type (episodic/semantic/procedural), a timestamp, and an importance — so you can scope retrieval (only this user) and score by recency/importance (next section). And you over-fetch (k=20) cheap candidates by similarity, then re-score and rerank down to the few you actually inject. What vector memory can't do: answer relational questions ("who reports to whom?", "what caused the outage?") — for that you need a graph, which is why production stacks go hybrid.

Summary Memory — Store as Compressed Text

The other half of the title. Instead of (or alongside) storing every raw memory, you summarize: keep a compressed representation that fits more history into fewer tokens. You met this in L133 (Short-Term vs Long-Term Memory) as a window-management tactic; here it's a storage strategy. The flavors (from the survey):

Rolling summary — one running summary of everything older than the recent turns (LangChain's ConversationSummaryMemory).
Hierarchical summaries — summaries of summaries, at multiple granularities (turn → session → week), so you can zoom in or out.
Task-conditioned compression — the current query decides which parts of history keep full detail and which get compressed.

Summary memory is transparent (it's just readable text), needs zero infra, and is wonderfully compact. Its fatal flaw has a name: summarization drift — each compression pass silently discards low-frequency details. Summarize the summary enough times and the exact $50k budget becomes "a budget," then vanishes. The rule of thumb: summarize the bulk for gist, but keep the exact, must-recall facts as discrete records (vector or structured) so they survive. Summary for breadth; vectors/structure for precise recall.

Retrieval Is More Than Similarity — The Score

Here's the idea that separates memory retrieval from document retrieval, and it's worth memorizing. Plain RAG ranks by similarity alone. But the most-similar memory isn't always the most useful one — a stale, trivial memory can be very "similar," while a fresh or critical memory is what you actually need. The Generative Agents paper (Park et al. 2023) solved this with a three-signal score that's now the field standard:

score = α·relevance + β·recency + γ·importance

relevance — cosine similarity between the query and the memory (how related?)

recency — an exponential decay over time since last access (how fresh?)

importance — an LLM-assigned 1–10 (how much does this matter? — mundane vs core)

Normalize each to [0,1], combine, take the top-k. The survey calls this a "substantial improvement over pure cosine similarity" — and it's intuitive: a person recalling "what should I cook?" weighs a recent comment, a strong preference, and relevance all at once. In code it's tiny:

# Agent retrieval is MORE than similarity (Generative Agents, Park et al. 2023):
def score(mem, query, now):
    relevance  = cosine(embed(query), mem.embedding)          # related to the situation
    recency    = 0.995 ** hours_since(mem.last_access, now)   # exponential decay → fresh wins
    importance = mem.importance / 10                          # LLM-rated 1–10 (core vs mundane)
    return relevance + recency + importance                   # (normalize each, then weight)

top = sorted(memories, key=lambda m: score(m, query, now), reverse=True)[:K]
context += [m.text for m in top]                              # inject only the top-K

Tune the weights to the job: a support agent leans on recency (the latest ticket state); a safety-critical agent leans on importance (never drop the allergy). Feel it below — flip a query and add the signals one at a time:

See It — Score & Retrieve

A store of memories, a situation (the query), and a scoring mode. Watch which top-3 memories get injected as you go from relevance-only to the full relevance + recency + importance blend:

A store can hold thousands of memories, but only the top few fit the context window — so retrieval has to choose well. Pick the agent’s situation (the query) and a scoring mode. RELEVANCE ONLY is naive RAG: it grabs the most semantically similar memories — including the stale, low-stakes “skips lunch.” Add RECENCY and IMPORTANCE and the recent, important “deadline” climbs in over it, while trivia like “favourite colour” (old, unimportant, irrelevant) never surfaces under any mode. This is the Generative Agents retrieval score — relevance (cosine similarity) + recency (exponential decay) + importance (a self-rated 1–10) — a substantial improvement over similarity alone. Watch the list re-sort as you add each signal.

The lesson in the re-sort: similarity gets you into the right neighborhood; recency and importance decide the final cut. A pure-similarity agent grabs the most related memory even if it's stale and trivial; the three-signal score surfaces what's related, fresh, and matters — and quietly buries the noise (the "favourite colour" memory never makes it).

The Full Retrieval Pipeline

Put it together. Production memory retrieval is a multi-stage pipeline — and you've seen every stage before in RAG, now tuned for memory:

Formulate the query. The raw user turn is often a poor retrieval query. Reformulate it with an LLM, fan out into multiple sub-queries, or use the agent's current sub-goal as the signal.
Search + filter. Dense (embeddings) + sparse (BM25) + metadata filters (user_id, type, time range) — over-fetch a candidate set. (Hybrid search + RRF, straight from the RAG section.)
Score. Re-score candidates by relevance + recency + importance (the memory-specific step).
Rerank. A cross-encoder / reranker (Cohere, a fine-tuned model, or an LLM) re-orders the shortlist for precision before it goes in the window.
Gate. Decide whether to retrieve at all — Self-RAG-style gating skips retrieval for turns that don't need it, cutting latency.
Inject. Drop the final top-k into the context window — the working memory the model reasons over.

Notice steps 1, 2, and 4 are pure RAG (Container 2). Steps 3 (memory score) and 5 (gating) are the memory-flavored additions. Memory retrieval is RAG with a clock and a sense of what matters.

The Write Path — Curate What You Keep

Retrieval is only as good as what's in the store — so the write path matters as much as the read path. Dumping raw text into a vector DB gives you a noisy, contradictory mess. A well-designed write path (from the survey) curates:

Filter — reject low-signal records (don't store "ok thanks").
Canonicalize — normalize dates, names, units so the same fact has one form.
Deduplicate — merge overlapping memories instead of piling up near-duplicates.
Tag metadata — timestamp, source, type, and a confidence/importance for later scoring.
Consolidate — periodically distill episodic → semantic (L134's reflection): "user corrected the date format 3×" → "user prefers DD/MM/YYYY."

And memory's hardest problem lives here too: contradiction & staleness. When a new memory conflicts with an old one — "I'm vegetarian" then later "I had steak" — you need a policy: temporal versioning (prefer the newest), source attribution (a user's statement outranks the agent's inference), contradiction detection (flag conflicts), and periodic consolidation (retire stale entries). A memory that's "accurate until it changes, then confidently wrong" is worse than no memory — curation is what prevents it.

2026 — The Frameworks

You rarely build all this from scratch in 2026 — a mature ecosystem of memory layers ships the pipeline. The three you'll meet most:

Mem0 — a vector + knowledge-graph hybrid. A two-phase write (LLM extraction → conflict detection + graph update) and three scopes (user / session / agent). Lightweight, framework-agnostic; reports big token savings vs full-context.
Zep / Graphiti — a temporal knowledge graph blending BM25 + embeddings + graph traversal, with no LLM call at retrieval time (fast, cheap) — the most production-validated as of 2026.
Letta (the MemGPT lineage) — a stateful runtime with editable memory blocks: "core memory" always in the prompt, archival memory in Postgres/SQLite reached via agent tools (archival_memory_search).

They differ in the storage substrate (pure-vector vs graph vs blocks) and when the LLM is involved, but they all implement the same shape you just learned: curate on write → store in a (hybrid) index → score + rerank on read → inject the top-k. Pick by your need for relationships (graph), latency (no-LLM retrieval), or prompt-resident state (blocks).

🧪 Try It Yourself

Reason it through, then check with the lab:

Predict: in the lab on the dinner query, switch from Relevance only to All three. Which memory drops out, which climbs in, and why?
Your agent must never forget a user's allergy, even months later when it's barely "similar" to the current chat. Which signal protects it, and what should you set?
You're storing a user's exact account number for precise recall. Vector or summary memory — and why not the other?
Retrieval keeps returning a fact the user changed last week. Name two write-path fixes.
Which retrieval stages are just the RAG you already know, and which are memory-specific?

→ (1) "Skips lunch" (stale, low-importance) drops; the recent, important "deadline" climbs in — because recency + importance outweigh its lower raw similarity. Similarity alone would keep the trivial one. (2) Importance — assign the allergy a high importance (e.g., 9–10) so the score keeps it retrievable even when relevance and recency are low. (3) Structured/vector record, not summary — an exact account number is a must-recall precise fact that summarization drift would corrupt or drop; summaries are for gist, discrete records for precision. (4) Temporal versioning (prefer the newest record) + contradiction detection / consolidation (flag the conflict and retire the stale entry); source attribution helps too. (5) RAG you know: query reformulation, hybrid dense+BM25 search, metadata filtering, reranking. Memory-specific: the relevance+recency+importance score and retrieval gating, plus the dynamic write/consolidation path.

Mental-Model Corrections

"Memory retrieval = vector search." Vector is one store among four (summary, structured/graph, skill), and retrieval is a pipeline (search → score → rerank → gate), not a single similarity() call.
"Just rank by similarity (it worked for RAG)." For memory, the most-similar isn't the most-useful. Score by relevance + recency + importance — a fresh or critical memory should beat a stale, vaguely-similar one.
"Summaries are a great way to store everything." Summaries drift — each pass drops detail. Use them for gist/breadth; keep exact, must-recall facts as discrete vector/structured records.
"Throw every memory into the store." Garbage in, garbage retrieved. The write path must filter, canonicalize, dedup, tag, and consolidate — curation is half the system.
"Once stored, a memory is true forever." Memories go stale and contradict. You need temporal versioning, contradiction detection, and periodic consolidation, or you'll confidently retrieve wrong facts.
"It's all custom infrastructure." In 2026 you mostly assemble it — Mem0 / Zep / Letta ship the curate→store→score→rerank pipeline; you choose by relationships, latency, and state model.

Key Takeaways

Memory retrieval is RAG pointed at the agent's own past — same embeddings/search/rerank you learned in Container 2, plus a memory-aware score and a dynamic write path.
Storage menu (combine them): vector (similarity, scales, loses relations) · summary (compact, transparent, drifts) · structured/graph (relationships, needs schema) · skill library (executable). The hybrid is standard.
Retrieval is more than similarity — the Generative Agents score: relevance (cosine) + recency (exponential decay) + importance (LLM 1–10). Tune the weights to the task; it beats pure similarity.
The pipeline: formulate query → dense+BM25+metadata search (over-fetch) → score → rerank → gate → inject top-k. (Stages 1/2/4 are RAG; score + gate are memory-specific.)
The write path is half the system: filter, canonicalize, dedup, tag, consolidate (episodic→semantic), and handle staleness/contradictions (newest-wins, source attribution).
2026: assemble with Mem0 (vector+graph), Zep/Graphiti (BM25+embed+graph, no-LLM retrieval), or Letta (memory blocks + archival tools).
Next — L136: Hands-On — A Memory-Aware Agent — wire all of Section 5 into a working agent that remembers across sessions.