Multi-Turn Conversation & Session Management

Introduction

We've designed the single response — fast (L224 — Treating Latency as a Product Feature), streamed (L225 — Streaming UX (TTFT, Token-by-Token, Markdown Buffering)), trustworthy (L226 — Showing Your Work), and defensive (L227 — Defensive UX). But real products aren't single responses — they're conversations that go many turns and sessions that span days. This lesson designs that.

The one fact that explains everything in this lesson: the model is stateless. It remembers nothing between API calls. The smooth, continuous 'conversation' your users experience is an illusion you construct — by resending the entire history on every single turn. Once you internalize that, the whole design problem falls out of it: cost grows, the context window fills, quality degrades over long chats, and you need real machinery to manage sessions and memory.

In this lesson:

The illusion of memory — statelessness, resending history, and the cost that creates
Managing the growing window — truncate, sliding window, summarize, hybrid
Why more context isn't better — lost in the middle and context rot
Conversations are trees — editing and branching, non-destructively
Sessions & persistent memory — threads, resume, and remembering across sessions (with privacy)

Scope: this is the product/UX view of conversations. The deep context-engineering and agent-memory mechanics — Managing the Context Window (Selection & Compression), Short-Term vs Long-Term Memory, Memory Stores & Retrieval (Vector + Summary) — are their own lessons; we'll use them and point you there.

Infographic titled 'Multi-Turn Conversation & Session Management' from the Designing AI Product Experiences section. The core truth, shown as a banner: the model is STATELESS — it remembers nothing between calls, so a 'conversation' is an illusion you create by RESENDING the whole history every turn. That single fact drives three problems shown as panels. (1) COST & THE WINDOW: each turn resends system + full transcript, so tokens grow linearly per turn and quadratically over the conversation (the agentic tax) and eventually overflow the context window — prompt caching makes the repeated prefix cost about 10 percent. (2) MANAGE THE WINDOW: when it fills, pick a strategy — truncate (drops early turns: catastrophic forgetting), sliding window (keep last K: loses distant history), summarize (a running summary: lossy on details), or hybrid (recent buffer + running summary + retrieve old facts on demand: the production default). (3) MORE CONTEXT ISN'T BETTER: lost-in-the-middle — models attend best to the start and end and worst to the middle (over 30 percent worse), and over many turns context rot/drift sets in and reliability drops after about 5 turns, so curate a focused context and put critical information at the edges. A second idea: a conversation is a TREE, not a list — editing a message or regenerating forks a branch (non-destructive), and the chat you see is just the path from root to the selected leaf; a session is that tree plus where you're standing in it, persisted so you get undo, branch comparison, and resumable threads. And across sessions, persistent memory (ChatGPT-style) extracts durable user facts and reinjects them — with user control (review, delete, forget) and privacy care. Roadmap strip: latency (L224), streaming UX (L225), showing your work (L226), defensive UX (L227), feedback UI (L228), multi-turn (L229), agent UX (L230). Takeaway banner: the model is stateless — you engineer the memory; manage the growing window (summarize/retrieve, not just truncate), keep context focused (lost-in-the-middle), treat conversations as branchable trees, and persist sessions with user-controlled memory.

The Conversation Is an Illusion You Maintain

When you chat with an LLM, it feels like the model is following along. It isn't. Each API call is independent — the model sees only what you put in that request and forgets it the instant it responds. To fake continuity, you (the application) store the transcript and resend all of it every turn:

turn N's request = system prompt + turns 1…N-1 (the whole history) + the new message + room for the reply

Two consequences fall straight out of this:

Cost grows super-linearly. Turn 2 resends turn 1; turn 10 resends turns 1–9; turn 40 resends 1–39. Tokens per turn grow linearly, so total tokens over the conversation grow quadratically — the "agentic tax" that makes long chats and multi-step agents surprisingly expensive.
The window is finite. That resent history has to fit in the model's context window alongside room for the output — and a long enough conversation won't fit.

The first lever is cost, and it's mostly free: because you resend the same prefix every turn, prompt caching (see Prompt Caching) lets the model reuse it at roughly ~10% of the input price. Always cache the stable prefix (system + early history) for any multi-turn or agent workload.

The second lever — fitting the history — is the heart of the lesson. Watch the window fill:

**Add conversation turns and watch the context window fill.** Because the model is **stateless**, every turn resends the **whole** history (system + every prior message + room for the reply). Eventually it can't all fit — the bar spills past the **context limit** and the oldest turns must be dropped. That overflow is the problem the rest of the lesson manages — and notice the cost: you're re-sending turns 1…N-1 on **every** new turn.

Managing the Growing Window

Once the conversation can't fit, you must decide what to keep. There's a spectrum of strategies, each a different trade between coherence and cost:

Strategy	What it does	The cost
Truncate	drop the oldest turns when you overflow	catastrophic forgetting — it loses turn 1 (the user's name, the original goal)
Sliding window	always keep the last K turns	cheap & coherent for recent context, but distant history is gone
Summarize	compress old turns into a running summary (e.g. at ~70–80% capacity)	a middle ground, but lossy — nuance, exact numbers, and constraints get smoothed away
Hybrid (production default)	recent turns verbatim + a running summary + retrieve old facts on demand (RAG over the transcript)	best coherence; more moving parts

The hybrid is what serious products run: keep the last K turns exact (recency matters most), a summary of everything older for gist, and a vector store of the full transcript so a relevant old fact can be pulled back in when this turn needs it. An importance score (recency + relevance + named entities) decides what's worth the tokens. Building that next-turn prompt looks like:

# The model is STATELESS — every turn you REBUILD and RESEND the whole prompt.
def build_prompt(history, user_msg, summary, store):
    return [
        {"role": "system", "content": SYSTEM},                       # always first — edges are attended best
        *([{"role": "system", "content": f"Summary so far: {summary}"}] if summary else []),  # gist of old turns
        *retrieve_relevant(store, user_msg),                         # hybrid: pull old facts back in on demand
        *history[-K:],                                               # sliding window: last K turns, verbatim
        {"role": "user", "content": user_msg},                       # the new turn — recent = well-attended
    ]
# No caching → turn N resends turns 1..N-1 → O(N²) tokens over the chat.  Cache the stable prefix → ~10%.

And tell the user when context limits bite — "this chat is long; I may have lost earlier details — want me to summarize?" is far better than silently forgetting their name. (The deep mechanics of selection & compression are Managing the Context Window (Selection & Compression) and Memory Stores & Retrieval (Vector + Summary).)

More Context Isn't Better — Lost in the Middle

It's tempting to think "just use a model with a huge context window and stuff everything in." That backfires, for two well-documented reasons:

Lost in the middle. Models attend best to information at the start and end of the context, and worst to the middle — recall can drop >30% for facts buried in the middle, even in long-context models. A bigger window doesn't fix this; it gives you a bigger middle to get lost in.
Context rot / drift. Over many turns, the conversational state degrades: the model makes premature assumptions, over-relies on its own earlier (possibly wrong) replies, and loses the thread — studies find reliability dropping sharply after only ~5 turns.

So the goal isn't maximum context — it's focused, well-ordered context:

Curate, don't dump. A short, relevant context beats a giant noisy one. This is why the hybrid strategy (retrieve only what's relevant) often outperforms stuffing the whole history in.
Put the critical stuff at the edges. System instructions and the current task early; the most relevant retrieved facts and the latest user message late. Don't bury the one thing that matters in the middle.
Re-state, don't assume. For long chats, periodically re-surface the key constraints (in the summary) so they stay at an attended position instead of rotting in the middle.

The counter-intuitive headline: a long context window is a budget, not a goal. Filling it is how chats get slower, costlier, and dumber. Spend it on the few things this turn actually needs.

Conversations Are Trees, Not Lists

Now the session side. We picture a chat as a straight line of messages — but the moment a user can edit an earlier message or regenerate a reply, it stops being a list and becomes a tree. Editing turn 3 doesn't overwrite it; it forks a new branch from that point, and the original branch still exists. The conversation you see is just the path from the root to the currently selected leaf.

This model (a tree of message nodes, each with a parent; the rendered chat = root→selected-node path) is exactly how ChatGPT-style edit / regenerate work — as non-destructive versioning, like save-points you can explore without losing your place. Try it:

**A conversation is a tree, not a list.** **Edit** one of your messages or **regenerate** a reply — it forks a new **branch**, and the original is never lost. The chat on the left is just the **path** from the top to the selected leaf; the tree on the right shows every branch (green = the path you're viewing). Use the **‹ n/m ›** switchers to jump between forks. This is exactly how edit/regenerate work in real products: **non-destructive versioning** — and a **session** is this whole tree plus where you're standing in it.

Two design rules the lab makes obvious: never destroy on edit (fork a branch — the user's original thread is sacred), and make branches visible (a tree, or at least a "‹ 1/2 ›" switcher at fork points) — hidden branches confuse and get lost. Give users an escape hatch to experiment without anxiety.

Sessions & Persistence

If a conversation is a tree, a session is that tree plus where you're standing in it — and since the model is stateless, persisting it is entirely your job. Get this right and a lot of features come for free:

The server stays stateless; you own the store. The conversation lives in your database (the message tree + a pointer to the active leaf), not in the model. Each turn you read it, build the prompt, and write the new nodes back.
Persisting the tree (not just the latest line) gives you undo, branch comparison, and resumable threads — close the tab, come back tomorrow, switch devices, and the session (and all its branches) is intact.
Resume cleanly. Re-open a thread → rehydrate the path → continue. This is the multi-turn cousin of the streaming resumability from L225 (Streaming UX (TTFT, Token-by-Token, Markdown Buffering)).
Name and organize threads. Long-lived assistants need thread lists, titles, and search — the session is a first-class object users return to, not a throwaway.

Design principle: the transcript tree is the source of truth, the model is a stateless function you call against it. Build your data model around that and multi-device, undo, and branching stop being special cases.

Persistent Memory — Remembering Across Sessions

Within a session you replay the transcript. But users also expect the assistant to remember them across sessions — your name, your preferences, that you're allergic to peanuts — without re-reading every past conversation. That's persistent memory: durable facts extracted from conversations and stored, then reinjected into the context of future sessions (the ChatGPT-style "memory" feature).

It's powerful and it's delicate — the design is mostly about control and trust:

Extract sparingly, the right things. Save stable, useful facts (preferences, recurring context) — not the whole transcript. (This is the human-in-the-loop / data-flywheel signal from L228 — Feedback UI & Human-in-the-Loop (The Data Flywheel) — turned toward the individual user.)
Make it inspectable and editable. Users must be able to see what's remembered, why a memory was used for a response, and delete or correct it. "Remember this" / "forget that" should just work.
Be careful with sensitive data. Don't auto-remember health, financial, or other sensitive details unless explicitly asked; honor consent and privacy. A memory feature that surfaces the wrong thing at the wrong moment is creepy and trust-destroying.
Show your work here too (L226 — Showing Your Work): surfacing "using your saved preference: vegetarian" lets the user catch and fix a wrong memory.

The deeper memory architecture — short-term vs long-term, episodic/semantic/procedural, vector + summary stores — is the agent-memory track (Short-Term vs Long-Term Memory, Memory Stores & Retrieval (Vector + Summary)). For product design, the rule is simple: remember usefully, show what you remember, and let the user stay in control.

🧪 Try It Yourself

Work these through, using the two labs above:

In the context-window lab, keep adding turns. What exactly overflows, and why does that imply your token cost is growing every turn even before you hit the limit?
A user told your assistant their name in turn 1; by turn 30 it's forgotten it. Which window strategy caused that, and what two strategies would have kept the name?
Your chat uses a 200K-context model and stuffs the entire history in every turn, yet it still misses details from the middle of long chats. Why — and what do you change?
In the branching lab, you edit an earlier message. What happens to the original conversation, and what is the chat you now see actually showing?
You're adding cross-session memory. Name two things you must let the user do, and one kind of data you should not auto-remember.

→ (1) The resent history (system + all prior turns + reply reserve) overflows the context window; because you re-send turns 1…N-1 on every turn, input tokens grow each turn → O(N²) total (mitigate with prompt caching on the stable prefix). (2) Truncation (or a too-small sliding window) dropped the oldest turn. Summarization would keep the name in the running summary (possibly lossy), and hybrid (retrieve from the full transcript) would pull it back exactly. (3) Lost in the middle — models attend worst to the middle, so a bigger window just means a bigger middle to lose facts in. Curate (retrieve only what's relevant), put critical info at the edges, and keep context focused, not maximal. (4) Nothing is destroyed — the edit forks a branch; both versions live in the tree, and the visible chat is just the path from root to the selected leaf. (5) Let the user see what's remembered and delete/correct it (and turn memory off); don't auto-remember sensitive data (health, finances) without explicit consent.

Mental-Model Corrections

“The model remembers our conversation.” It's stateless — it remembers nothing. You resend the whole history every turn; the 'memory' is your application's.
“Longer chats are basically free.” Resending history makes cost grow O(N²) over the conversation. Cache the prefix (~10%) and manage the window.
“Just truncate when it's full.” Truncation = catastrophic forgetting of early turns. Prefer summarize + retrieve (hybrid) so old facts can come back.
“A bigger context window solves it.” Lost in the middle + context rot mean more context is often worse. Curate a focused context; put the important parts at the edges.
“Editing a message rewrites the conversation.” It forks a branch — non-destructive. The chat you see is the path through a tree.
“The session is the latest message list.” It's the whole tree + your position in it — persist that and you get undo, branches, and resume for free.
“Persistent memory = store everything.” Remember sparingly and usefully, make it inspectable and deletable, and don't auto-remember sensitive data. Control and transparency are the feature.

Key Takeaways

The model is stateless — you engineer the memory. A conversation is an illusion you maintain by resending the whole history every turn, which makes cost grow O(N²) (cache the prefix → ~10%) and the context window eventually overflow.
Manage the growing window: truncate (forgets early), sliding window (recent only), summarize (lossy), or hybrid — recent verbatim + running summary + retrieve old facts on demand (the production default). Tell the user when limits bite.
More context isn't better: lost in the middle (best at the edges, >30% worse in the middle) and context rot after ~5 turns mean you should curate a focused context and put critical info at the edges, not maximize tokens.
Conversations are trees, not lists: edit / regenerate fork a branch (non-destructive); the visible chat is the path root→leaf. Make branches visible.
A session is the tree + your position — persist it (the model stays stateless; you own the store) for free undo, branch comparison, and resumable threads.
Persistent memory across sessions must be inspectable, editable, and privacy-careful — remember usefully, show what you remember, and keep the user in control.
Next — L230 (Agent UX: Showing Reasoning, Progress & Approvals) — the section finale: designing for agents that take minutes, use tools, and act on the world — showing their reasoning and progress, and gating their actions with approvals.