Memory, State & History as Context

Introduction

Last lesson you learned to keep the window small by selecting and compressing what's in it. But that raises an obvious question: where do the facts you dropped go? You can't just lose them — "what plan is this customer on?" might have been said 40 turns (or three sessions) ago.

The answer is memory: durable information kept outside the context window and pulled back only when it's relevant. This is the missing half of context engineering. Selection/compression manage the finite window; memory gives the system something boundless to draw from without paying for it on every token.

This lesson is the context-engineering view of memory — how it fits as a source you assemble. We'll go deep on memory stores, retrieval, and a full memory-aware agent in the Agents container (C3 §5); here we build the mental model and the loop.

In this lesson you'll learn:

The crucial distinction between history, state, and memory
The RAM/disk mental model: short-term (working) vs long-term memory
What long-term memory holds: episodic, semantic, procedural
The write → retrieve → inject loop — memory as a context source
The pitfalls (memory bloat, stale facts, leaking what you stored)

History vs State vs Memory

People lump these together; keeping them distinct is what makes the design click:

	History	State	Memory
What it is	the raw transcript of turns	the working scratchpad for the current task	durable facts worth keeping
Where it lives	in the context window	in the window (or app variables)	an external store, outside the window
Lifespan	this conversation	this task	across turns and sessions
Managed by	selection / compression (last lesson)	your app logic	the write/retrieve loop (this lesson)

A quick example. A user says "I'm on the Pro plan; help me cancel my June order."

History: the literal back-and-forth messages.
State: task = "cancel order"; order_month = "June" — scratchpad for this job, gone when it's done.
Memory: "This user is on the Pro plan" — worth remembering next week, so it goes to the store.

The RAM / Disk Mental Model

The cleanest way to hold all this in your head comes from MemGPT (now Letta): treat the LLM like a tiny computer.

The context window is RAM — fast, but small and ephemeral. It holds the short-term / working memory: system prompt, recent history, current state. It's wiped between sessions.
An external store is disk — vast and persistent. It holds long-term memory that survives across sessions.

Just like a real program, you page information between them: write important facts from RAM down to disk, and load the relevant ones back into RAM when a task needs them. You can't fit everything in RAM (context rot, overflow — last two lessons), so good systems keep RAM lean and lean on disk. In production this is often a three-tier stack: in-context working memory → a session summary of compressed facts → a long-term persistent store (typically vector + sometimes graph).

An infographic titled 'Memory: What the Window Forgets, the Store Remembers'. The top zone is short-term working memory, drawn as the context window (RAM) holding the system prompt, recent history, and current state or scratchpad, labelled fast, tiny, ephemeral, this session. The bottom zone is long-term memory (disk), an external store holding three buckets: episodic (past events), semantic (facts and preferences), and procedural (rules and how-to), labelled vast, persistent, across sessions. Between the two zones, a downward arrow labelled WRITE — extract and store durable facts, and an upward arrow labelled RETRIEVE the relevant few, then INJECT just-in-time. A bottom banner says the window is RAM, memory is disk: page facts out when done, pull the relevant few back when needed.

The diagram captures the whole loop: a small, fast window on top; a vast, persistent store below; and the two flows — write down, retrieve up — that connect them.

What Long-Term Memory Holds: Episodic, Semantic, Procedural

Cognitive science gives long-term memory a useful three-way split (you'll see these names everywhere):

Episodic — specific past events/interactions, usually with a timestamp. "On June 12, Jane reported a delayed refund." (A log of what happened.)
Semantic — facts and preferences; the agent's stable knowledge about the user/world. "Jane is on the Pro plan and prefers email." (What's true.)
Procedural — rules, skills, how-to; often the system prompt or stored guidelines. "Always greet returning customers by name; follow the refund SOP." (How to act.)

You don't need all three for every app — a support bot might just keep semantic facts about each customer — but knowing the categories helps you decide what's worth storing and how you'll retrieve it. (Full treatment, plus how to store and retrieve each, in C3 §5.)

The Memory Loop: Write → Retrieve → Inject

Memory is a cycle you run around the model, made of a few operations: store, retrieve, update, summarize, discard. The core loop:

Write — after a turn (or when compacting the window), extract the durable facts and store them. Not the whole transcript — the facts worth keeping.
Retrieve — on a new turn, fetch the few memories relevant to the current message (by keyword, or — properly — by embedding similarity; that's the RAG machinery in Container 2).
Inject — drop those retrieved facts into the context you assemble, just-in-time.

Here's the whole idea in deterministic code — a tiny store you can run (real systems swap the keyword match for embedding search):

# Long-term memory = a store OUTSIDE the context window: write facts, read them back.
MEMORY = {}

def remember(key, fact):
    MEMORY[key] = fact                       # store / update

def recall(query):                           # naive keyword match
    q = query.lower()                        # (real systems retrieve by EMBEDDING — see RAG)
    return [fact for key, fact in MEMORY.items() if q in (key + " " + fact).lower()]

remember("plan",  "Jane is on the Pro plan (since 2024).")
remember("pref",  "Jane prefers email over phone.")
remember("issue", "Jane had a refund delayed 3 weeks in June.")

# A NEW session: the chat history is gone, but MEMORY persists.
# Pull back only what's relevant to this turn — not everything:
print(recall("refund"))   # -> ['Jane had a refund delayed 3 weeks in June.']
print(recall("Jane"))     # -> all three facts about Jane

And retrieval feeds straight back into the context you assemble (the through-line of this whole section):

# Memory plugs straight into the context you assemble (recall the last two lessons):
def build_context(system, user_msg):
    facts = recall(user_msg)                 # RETRIEVE the relevant memories...
    memory_block = "\n".join(f"- {f}" for f in facts)
    return "\n\n".join([
        f"## System\n{system}",
        f"## What we know about the user\n{memory_block}",   # ...and INJECT them
        f"## User\n{user_msg}",
    ])

print(build_context("You are a support agent.", "I'm still waiting on my refund"))

That's memory as a context source: the window stays small, but the system behaves as if it remembers everything — because the relevant slice is paged in on demand.

Memory as a Context Source (and the Tools)

Step back and the section's arc closes. Context engineering (#1) said the window is assembled from sources and should hold the smallest high-signal set. Window management (#2) keeps that set small via selection + compression. Memory is what makes that safe — you can drop and compress aggressively because the important facts are saved on "disk" and retrievable.

You rarely build this from scratch in production. Two common approaches:

Mem0 — a bolt-on memory layer: add() to store, search() to retrieve; it embeds facts into a vector DB under the hood. You add it to whatever framework you already use.
Letta (MemGPT) — an OS-style agent runtime where the model itself decides what to promote into context (RAM) and evict to archival memory (disk), via memory-edit tools.

Both implement the same write/retrieve loop you just saw. We build a memory-aware agent end-to-end in C3 §5; for now, the mental model is the lesson.

Pitfalls & Common Mistakes

Memory bloat. Storing every message means retrieval returns noise. Store distilled facts, not raw transcripts — and discard what stops being useful.
Stale / contradictory memory. Append-only memory rots: "Pro plan" stays after the user downgrades. You need update and expiry, not just store.
Retrieving irrelevant memories. Injecting the wrong "facts" poisons the context exactly like any low-signal tokens (context engineering, #1). Retrieve the few relevant, not everything.
Confusing state with memory. Task scratchpad is throwaway; don't persist it as long-term memory (and don't make durable facts live only in fragile state).
Storing sensitive data. Memory often holds PII and preferences — a privacy and security surface. What you persist can later leak (the last lesson of this section) or be poisoned by a malicious input. Store deliberately.

🧪 Try It Yourself

Use the little memory store above. Add three facts about a user (one episodic — a dated event; one semantic — a preference; one procedural — a rule for how to treat them). Then simulate a new session: imagine the chat history is wiped, and call recall(...) with a fresh user message — confirm you get back only the relevant fact(s), not all of them.

Now the judgment call that is memory engineering: from a real conversation, list five things said and mark each store (durable) or discard (throwaway). What makes the cut — and what would you regret keeping a month from now?

Memory Loop — from a first session, choose which facts to write to the store: the semantic facts (on the Pro plan, prefers email), the episodic event (a June 12 delayed refund), and a throwaway noise fact (it was raining). Then a new session wipes the context window — the chat history is gone — and a fresh user question retrieves only the relevant few stored facts and injects them just-in-time. Ask about priority support and the agent answers correctly only if you stored the Pro-plan fact; discard it and the agent forgets and has to ask. Storing the noise fact flags memory bloat. Makes concrete the write → retrieve → inject loop and memory as a context source: a small window that behaves as if it remembers everything because the durable facts live outside it, on disk.

Key Takeaways

Memory is durable information kept outside the window and pulled back when relevant — the boundless complement to the finite window.
Keep history (transcript, in-window), state (task scratchpad), and memory (durable, external) distinct — they have different lifespans and are managed differently.
The RAM/disk model (MemGPT/Letta): the window is small fast RAM; the store is vast persistent disk; you page facts between them.
Long-term memory splits into episodic (events), semantic (facts/prefs), procedural (rules/how-to).
The loop is write → retrieve → inject (store/retrieve/update/summarize/discard) — memory plugs in as a context source, letting you keep the window small without forgetting.
Watch for bloat, stale facts, and the privacy surface of what you persist.

Next: we turn to the defensive half of this section — Prompt Injection & Jailbreaking — because everything you put in the context (including retrieved data and memory) is also an attack surface.