Skip to main content
CodePeetCodePeet
AI Agents & MCP

Short-Term vs Long-Term Memory

Introduction

Last lesson you learned why agents need memory: an LLM is stateless, and the context window is working memory, not a long-term store. This lesson splits that idea into the two tiers every real memory system is built on — and shows how they work together.

Short-term memory is what the agent is thinking about right now. Long-term memory is what it keeps. One is a small, fast, expensive window; the other is a vast, durable store.

This isn't a new idea — it's how human memory is organized, and it's the single most important architectural distinction in agent memory. Get the boundary right and your agent is fast, cheap, and coherent over months. Get it wrong and you either blow your token budget or build another amnesiac.

In this lesson:

  • The two tiers — short-term (working / context window) vs long-term (durable store), and why they're separated
  • Managing short-term when the window fills — sliding window, summarization, offloading — felt in a hands-on lab
  • The interplay — how memories move up (retrieve) and down (consolidate) between the tiers
  • What goes where, the 2026 reality, and the traps

Scope: this is the two-tier split and how to manage each tier. The kinds of long-term memory (episodic / semantic / procedural) get their own deep-dive next (L134 — Episodic, Semantic & Procedural Memory), and the store + retrieval mechanics (vector search, summaries) follow in L135 (Memory Stores & Retrieval).

An infographic titled 'Short-Term vs Long-Term Memory' showing an agent's memory hierarchy as two stacked tiers. The top tier is SHORT-TERM, or WORKING memory — the context window, the model's RAM. It is in-context and fast but bounded (roughly 8k to 1M tokens), volatile (gone when the session ends), and re-sent and re-billed on every call, so it is the hot path. It holds the pinned system prompt, the recent turns, the current task state, and a scratchpad of tool results. The bottom tier is LONG-TERM memory — an external durable store, the agent's disk. It is persistent across sessions, effectively unbounded, and retrieved on demand, which decouples it from inference cost, so it is the cold path. It holds user preferences, past conversations, durable facts, and learned procedures, and in production is a vector database or knowledge graph. Between the two tiers run two arrows that make the hierarchy work: RETRIEVE is synchronous and points up — it pulls the relevant memory from the store into the working context right when it is needed; CONSOLIDATE is asynchronous and points down — a background worker extracts the valuable facts from the conversation and writes them out to the store. This mirrors the classic Atkinson-Shiffrin model of human memory, where sensory input passes into a small short-term store (about 7 items, lasting seconds) and, through rehearsal and consolidation, into a vast and durable long-term store, with retrieval bringing it back. Because the window is finite, short-term memory must be managed when it fills: keep everything and it overflows; use a sliding window of the last k turns and it stays cheap but drops older turns (and you must pin the system prompt); summarize old turns into a running summary and you trade size for fidelity, keeping the gist but losing exact detail; or offload durable facts to long-term and retrieve them on demand, keeping the window small and recall exact. The infographic's three cards cover the short-term tier (a bounded hot path you actively manage), the long-term tier (a durable store you write to and retrieve from), and the interplay (retrieve synchronously, consolidate asynchronously, and always pin the system prompt). The 2026 note: a bigger context window does not remove the need for long-term memory, because cost stays linear and attention degrades in the middle (lost in the middle) — the separation of ephemeral context from a durable store is what lets an agent feel coherent over time. Takeaway: short-term is the bounded working window you manage; long-term is the durable store you persist to and retrieve from; together they form the agent's memory hierarchy.

Two Tiers, One Mind

Here is the whole lesson in one table. Every agent memory system is some arrangement of these two tiers:

Short-term (working)Long-term
Isthe context windowan external store (vector DB / graph)
OS analogyRAMdisk
Pathhot — synchronous, in every inferencecold — async, retrieved on demand
Lifetimevolatile (this session)persistent (across sessions)
Sizebounded (~8k–1M tokens)effectively unbounded
Costre-billed every turndecoupled from inference cost
Holdssystem prompt, recent turns, task statepreferences, past events, facts, skills

The crucial word is decoupled. As one 2026 engineering guide puts it: "Every byte of short-term memory you use costs you money on every inference pass." Long-term memory breaks that — "infinite storage for background details, but at the cost of retrieval complexity." You keep the working set tiny and pull from the store only what this turn needs. In code, the two tiers look completely different:

# SHORT-TERM (working memory) = the messages you re-send every call. The HOT path.
messages = [system_prompt, *recent_turns, user_msg]      # bounded by the context window
reply = client.messages.create(model="claude-opus-4-8", messages=messages)
#   in-context, fast — but volatile, finite, and re-billed on every single call.

# LONG-TERM = a durable store you write to and retrieve from. The COLD path.
store.write(user_id, {"budget": 50_000})                 # async; persists across sessions
facts = store.search(user_id, query=user_msg, k=3)       # pull back only what's relevant
messages = [system_prompt, *facts, *recent_turns, user_msg]   # inject into working memory
#   persistent, ~unbounded — decoupled from inference cost; retrieved on demand.

That's the skeleton of every memory-aware agent: a small, hot working set you send every call, and a large, cold store you read from and write to around it. The rest of this lesson is about the boundary between them — how to keep the window from overflowing, and how memories cross from one tier to the other.

The Human Blueprint

This two-tier design isn't an engineering accident — it's the Atkinson–Shiffrin model of human memory (1968), the most influential model in cognitive psychology, ported almost verbatim to agents:

  • Sensory memory → raw input (what you just saw/heard), held for under a second. For an agent: the raw tool outputs and user input arriving this turn.
  • Short-term / working memory → a tiny buffer — famously 7 ± 2 items (Miller's "magic number"), held for seconds. For an agent: the context window. Small, fast, and where the actual thinking happens (Baddeley later refined this into "working memory").
  • Long-term memory → a vast, durable store of facts and experiences, kept for years. For an agent: the external memory store.

The two transfers are exactly the ones your agent needs: information enters short-term via attention, and crosses into long-term via rehearsal / consolidation — deliberately deciding "this is worth keeping." Retrieval brings it back. When you build an agent's memory, you are re-implementing a 50-year-old cognitive architecture: a small working store, a large durable store, and disciplined movement between them.

Short-Term Memory: The Context Window Up Close

Short-term memory is the easy one — you've been using it the whole course. It is the context window: the messages array you send on every call. Everything the model can directly "see" lives here:

  • the system prompt (its instructions and persona),
  • the recent conversation turns,
  • the current task state — the half-finished plan, the variable it's tracking, what "it" refers to,
  • a scratchpad of recent tool results and reasoning.

Its strengths are real: it's immediate (no retrieval step — the model attends to it directly) and high-fidelity (the exact tokens, not a lossy summary). But it has three hard limits you now know by heart: it's bounded (a fixed token budget), volatile (gone at session end), and expensive (re-sent and re-billed on every turn). Which leads to the one problem that defines short-term memory engineering: it fills up. A long conversation, a big tool result, a multi-hour task — sooner or later the history won't fit. So the real skill isn't using short-term memory; it's managing it.

When the Window Fills — Managing Short-Term

When the history won't fit the window, you have four moves — each trading window space for recall fidelity:

  1. Keep everything. Simplest, highest-fidelity — until it overflows. Then the call errors, or the framework silently drops the oldest turns, and you're paying to re-send a huge context every turn (and inviting lost-in-the-middle). Fine for short chats; a trap for long ones.
  2. Sliding window — keep only the last k turns (LangChain's ConversationBufferWindowMemory). Cheap and bounded, but it forgets anything older. ⚠️ Always pin the system prompt — "blindly chopping off the start of a conversation deletes the system prompt."
  3. Summarize — compress old turns into a running summary (ConversationSummaryMemory); typically triggered near 70–80% capacity. This is the key shift: from managing size to managing fidelity. You keep the gist forever in a few tokens — but you lose exact detail.
  4. Summary-buffer hybrid — the production default: recent turns verbatim (full fidelity where it matters) + older turns summarized (gist for the rest). ConversationSummaryBufferMemory.

And the fifth, which graduates into the other tier: offload the durable facts to long-term and retrieve them on demand. In code:

# When working memory overflows, you MANAGE it. Three classic moves:

# 1) SLIDING WINDOW — keep only the last k turns. Cheap, but it FORGETS older ones.
working = [system_prompt] + history[-K:]          # ALWAYS pin the system prompt!

# 2) SUMMARIZE — compress old turns into a running summary (manage fidelity, not size).
if tokens(history) > 0.8 * WINDOW:                # trigger near ~80% capacity
    summary = summarize(history[:-K])             # keeps the gist, loses exact detail
    working = [system_prompt, summary] + history[-K:]

# 3) OFFLOAD — write durable facts out, retrieve on demand (this is the long-term tier).
store.write(user_id, extract_facts(history))      # async consolidation
working = [system_prompt, *store.search(user_id, q=user_msg), *history[-K:]]

Each move loses something different — and the only one that keeps the window small and preserves an exact fact is the last one. Feel the difference yourself:

See It — Manage the Window

It's turn 13 of a long chat and the context window is full — but back in turn 1 the user set a $50k budget. Pick a strategy and run the recall test:

It’s turn 13 of a long chat and the context window can’t hold everything — yet back in turn 1 the user set a $50k budget. Pick how to manage short-term memory, and the recall test “what budget did I set?” exposes each trade-off. KEEP ALL recalls $50k but the window OVERFLOWS (re-billed every turn, lost-in-the-middle). SLIDING WINDOW fits, but it dropped turns 1–8 so the budget is LOST (and never drop the pinned system prompt). SUMMARIZE fits and keeps the gist, but the exact $50k was compressed away (gist only). OFFLOAD → long-term writes the fact to a durable store and retrieves it on demand: the window stays small AND recall is exact. The lesson lands — short-term is a bounded hot path you must actively manage; long-term is the durable store you write to and retrieve from.

The pattern to internalize:

  • Keep all → recalls it, but the window overflows (cost + lost-in-the-middle).
  • Sliding window → fits, but dropped the budget with the old turns — lost.
  • Summarize → fits and keeps the gist ("there's a budget") but loses the exact $50k.
  • Offload → long-term → fits and recalls exactly, because the fact was written to the store and retrieved on demand.

That last column is the entire reason long-term memory exists — and the bridge to the rest of this section.

Long-Term Memory: The Durable Store

Long-term memory is "the externalization of state into a durable storage system." It lives outside the context window — in production, a vector database or knowledge graph — and it inverts every property of short-term memory:

  • Persistent, not volatile — it survives across sessions, restarts, and weeks of silence.
  • ~Unbounded, not capped — store a year of interactions; you're limited by disk, not by a token budget.
  • Cost-decoupled, not re-billed — it sits idle and cheap until you retrieve from it, so it never inflates your per-call token bill.
  • Selective, not all-or-nothing — you pull back the few relevant items, not the whole history.

The price you pay for those gains is retrieval complexity: unlike short-term memory, the model can't see long-term memory directly. Something has to fetch the right piece and place it into the working window at the right moment — and the quality of that fetch makes or breaks the agent. Long-term memory also comes in kinds — episodic (events), semantic (facts), procedural (skills) — and is stored and searched with specific machinery. Both of those are deep enough to be their own lessons: the kinds are L134 (Episodic, Semantic & Procedural Memory), and the store + retrieval mechanics are L135 (Memory Stores & Retrieval). Here, hold one idea: long-term memory is a durable store you write to and retrieve from.

The Interplay — Retrieve & Consolidate

The tiers aren't independent — they form a hierarchy, and memories move between them in two directions. This is the heart of the system:

  • Retrieve (long-term → short-term) — synchronous, on the hot path. Before the model answers, you query the store for what's relevant to this turn and inject it into the context window. ("User is asking about budget → fetch budget=$50k → put it in the prompt.")
  • Consolidate (short-term → long-term) — asynchronous, off the hot path. After (or alongside) the turn, a background worker scans the conversation, extracts the durable facts, and writes them to the store — the "this is worth keeping" step. ("User stated a budget → save it for later.")

This is the write-through cache pattern: the working window is your fast cache; the store is your durable backing. Retrieval is a cache read; consolidation is a cache write-back. It's also exactly MemGPT's OS analogy — the agent pages information between RAM (context) and disk (store), moving things in when needed and out when not. The big shift the whole field made: stop trying to fit everything in one giant window; build a hierarchy and move memories across the boundary.

Working memory holds what's active; long-term holds what's kept. Retrieve pulls the right thing up into focus; consolidate writes the important thing down for later. That loop is an agent's memory.

What Goes Where

The one judgment call this lesson trains: what belongs in which tier? The rule of thumb — short-term for what's active now; long-term for what must outlive this turn:

Put in short-term (window)Put in long-term (store)
The system prompt & current instructionsDurable user preferences ("vegetarian")
The last N turns of dialoguePast events / sessions ("ordered X last week")
Current task state, the active planStable facts & domain knowledge
Reference resolution (what "it" means)Learned procedures / skills
This turn's tool results (scratchpad)Anything needed across sessions

Two cheap heuristics make the call: (1) Will I need this after the session ends? If yes → long-term. (2) Do I need it every turn, or only sometimes? Every turn → keep it resident in short-term; only sometimes → store it and retrieve on demand. Get this division right and the agent stays cheap (small window), fast (little to retrieve), and coherent (nothing important is lost).

The 2026 Reality

Two things are true in 2026, and they don't cancel out.

(1) Context windows are huge — hundreds of thousands to millions of tokens. That genuinely expands short-term memory and removes a lot of aggressive trimming. (2) It still doesn't replace long-term memory — for the reasons you now know cold: a big window is still volatile (gone at session end), its cost stays linear (double the window, double the per-token bill), and it rots in the middle (lost-in-the-middle). As the guides put it, "this makes memory a finite resource that requires strict memory management" — and "the separation of ephemeral context from durable indices is the only way to build AI applications that feel coherent over time."

The platforms now ship both tiers as first-class tools. For short-term management, Claude offers context editing (auto-clears stale tool results from the window as it fills) and compaction (server-side summarization of older context near the limit) — your sliding-window and summarize strategies, automated. For long-term, the memory tool (/memories) persists across sessions. The modern stack is explicitly hierarchical memory — manage the hot window and keep a durable store — not "just use a bigger window."

🧪 Try It Yourself

Reason through these, then use the lab to confirm:

  1. Predict: in the lab, choose Sliding window and run the recall test. Why does it fail, and what one line of config would make it catastrophically worse?
  2. You're storing a user's current multi-step checkout state mid-conversation. Short-term or long-term — and why?
  3. Summarize keeps the conversation flowing under budget but the agent gives a vague answer about the $50k. What exactly was traded away?
  4. Your context window is now 1M tokens. Give two reasons you'd still build a long-term store.
  5. Name the two directions memory moves between tiers, and which one is synchronous.

→ (1) Sliding window keeps only the last k turns, so turn-1's budget was dropped — lost. The catastrophic config: a window that doesn't pin the system prompt, so truncation eventually deletes the agent's instructions too. (2) Short-term — it's active task state needed this turn and this session; it's transient (the checkout ends), so it belongs in the working window, not the durable store. (3) The exact detail ($50k) — summarization manages fidelity, not size, keeping the gist ("there is a budget") while compressing the precise number away. (4) A 1M window is still volatile (no cross-session persistence) and cost scales linearly + rots in the middle; long-term memory is persistent, cost-decoupled, and selective. (5) Retrieve (long-term → short-term, synchronous, on the hot path) and consolidate (short-term → long-term, asynchronous).

Mental-Model Corrections

  • "Short-term and long-term memory are two databases." They're two tiers of one hierarchy — short-term is the in-context window (RAM), long-term the external store (disk) — and memories move between them.
  • "Bigger context window = I don't need long-term memory." A window is still volatile, linearly costly, and rots in the middle. Long-term adds persistence + cost-decoupling + selectivity — different job.
  • "Just keep the whole history in context." It overflows, re-bills every token each turn, and buries facts mid-context. Manage it: window, summarize, or offload.
  • "Truncating old turns is safe." Naive truncation can delete the system prompt — pin it — and silently drops facts you'll need. Prefer summarize/offload for anything durable.
  • "Summarizing loses nothing important." It trades size for fidelity — you keep the gist and lose exact detail. Offload the exact facts you must recall precisely.
  • "Memory just means saving to long-term." Half of it is retrieval — pulling the right memory back into the window at the right moment. Writing without good reading is useless.

Key Takeaways

  • Two tiers, one hierarchy. Short-term (working) = the context window (RAM): hot, in-context, bounded, volatile, re-billed every turn. Long-term = an external store (disk): cold, persistent, ~unbounded, cost-decoupled, retrieved on demand.
  • It mirrors human memory (Atkinson–Shiffrin): sensory → short-term (7±2, seconds) → long-term (durable), with consolidation up and retrieval back.
  • Short-term's defining problem is that it fills up — manage it: keep-all (overflows), sliding window (drops old — pin the system prompt!), summarize (gist not detail — size→fidelity), summary-buffer hybrid, or offload to long-term.
  • Long-term is a durable store you write to and retrieve from — persistent, unbounded, selective; the price is retrieval complexity (the model can't see it directly).
  • The tiers interact via a write-through cache: retrieve (long→short, sync) pulls the relevant memory into context; consolidate (short→long, async) writes durable facts out. (MemGPT's RAM↔disk paging.)
  • What goes where: short-term = active now (system prompt, recent turns, task state); long-term = must outlive the turn (preferences, past events, facts, skills).
  • 2026: bigger windows expand short-term but don't replace long-term (still volatile, linearly costly, lost-in-the-middle). Claude ships context editing + compaction (short-term) and the memory tool (long-term) — hierarchical memory, not a bigger window.
  • Next — L134: Episodic, Semantic & Procedural Memory — the three kinds of long-term memory, in depth.