Managing the Context Window (Selection & Compression)

Introduction

Last lesson gave you the why of context engineering — context is a finite resource, and the goal is the smallest set of high-signal tokens. This lesson is the first and most fundamental how: when the stuff you'd like to include is bigger than the window, you have exactly two levers — decide what to include (selection) and shrink what you include (compression).

This isn't a niche concern. Every chatbot, every agent, every multi-turn anything hits it: the conversation keeps growing, but the window doesn't. Manage it well and the system stays coherent and cheap; manage it badly and it either overflows (hard error) or rots (silent quality collapse — last lesson's context rot).

In this lesson you'll learn:

Why the window fills up — and the two failure modes when it does
Selection: recency / sliding window vs relevance-based keeping
Compression: rolling summarization and compaction
How to assemble a budgeted window (system + summary + recent + output reserve)
The tradeoffs — when truncation is fine, and when it quietly destroys coherence

The Problem: The Window Always Fills Up

Here's the mechanic people miss: an LLM is stateless. To continue a conversation, you re-send the entire history on every single call. So the tokens you spend grow turn after turn — turn 50 carries all 49 before it. Two things eventually break:

Overflow — the request exceeds the window and the API errors out. Hard failure.
Context rot — long before overflow, recall degrades (last lesson). Soft failure: the model starts "forgetting" things that are technically still in the prompt.

Either way you can't just keep appending. You have to actively manage what stays in the window — which is what the rest of this lesson is about. (You'll feel this directly in the Try It Yourself widget below: add turns and watch the budget fill.)

Two Levers: Selection & Compression

Everything reduces to two moves:

Selection — what to include. You can't keep all of history, so you choose a subset: the most recent turns, the most relevant ones, and always the system prompt + the latest user message.
Compression — how to shrink what you keep. Instead of dropping old turns entirely, replace them with a much shorter summary that preserves the gist.

In practice you use both: keep the last few turns verbatim (recency), roll everything older into a running summary (compression), and reserve room for the answer. That hybrid is the workhorse of production chat and agents.

An infographic titled 'Managing the Context Window: Select & Compress'. On the left, a tall stack of many conversation turns overflowing a window outline, labelled grows every turn and overflows. In the middle, two operations: SELECT, meaning keep the recent and relevant turns and drop the rest; and COMPRESS, meaning summarize the old turns into a short summary. On the right, a managed context that fits the window, containing the system prompt, a summary of the older turns, the recent turns kept verbatim, and reserved space for the output. A bottom banner says keep the few high-signal tokens: recent verbatim plus the rest summarized; truncate blindly and you lose the thread, summarize and you keep it cheaply.

The picture above is the whole lesson in one frame: a raw history that overflows on the left, the two operations in the middle, and a managed context that fits on the right. Now the details of each lever.

Selection: Which Turns Make the Cut

Recency (a sliding window). The simplest selection: keep the last N messages, drop the rest. Cheap, predictable, no extra calls. It's fine for stateless-ish chat — but pure truncation destroys coherence on long tasks: drop turn 3 where the user stated their goal, and turn 50 wanders off-track.

Relevance-based selection. Smarter: keep the turns that matter to the current question, even if they're old — using semantic similarity to pull back the relevant earlier exchange. (This is retrieval applied to your own conversation; you'll build the machinery for it in the RAG container.) It costs a lookup but stops important early facts from falling off the end of the window.

Always keep the system prompt (the rules) and the newest user turn (the actual question). Most failures here are dropping one of those, or dropping the one old turn that held the key constraint.

Compression: Summarize Instead of Forget

Selection throws tokens away. Compression keeps the information but spends fewer tokens on it.

Rolling summarization. A common trigger: when you reach ~70–80% of the window, summarize the oldest turns into a compact paragraph and replace them with it. You keep condensed old history + full-fidelity recent turns.
Compaction. For long-running agents, summarize the whole trajectory as it nears the limit — preserve decisions, open questions, and key facts; discard redundant tool outputs and superseded reasoning — then continue from the compressed version. Anthropic has productized this as automatic, server-side context compaction within a session.
Document compression. When a single retrieved doc is huge, strip filler and redundancy before inserting it (often a 40–60% token cut) so it fits without losing the substance.

The cost: compression usually means an extra LLM call to produce the summary, and an over-aggressive summary can drop a detail you needed. Summarize the old stuff, keep the recent stuff exact.

Putting It Together: A Budgeted Window

Think of the window as a budget you allocate across parts: system + summary-of-old + recent-turns + a reserve for the output. The selection half is pure bookkeeping you can write today — keep the newest turns that fit, mark the rest for summarizing. Run this (no API needed):

# SELECTION by budget: keep the system prompt + the most RECENT turns that fit.
# (COMPRESSION — summarizing what we drop — needs a model call; see below.)
def trim_to_budget(system_tokens, turns, budget, output_reserve):
    kept, used = [], system_tokens + output_reserve
    for label, tok in reversed(turns):          # walk newest -> oldest
        if used + tok > budget:
            break                                # no room for older turns
        kept.append((label, tok))
        used += tok
    kept.reverse()                               # back to chronological order
    dropped = turns[: len(turns) - len(kept)]
    return kept, dropped, used

turns = [(f"turn {i}", 300) for i in range(1, 21)]      # 20 turns, ~300 tokens each
kept, dropped, used = trim_to_budget(
    system_tokens=400, turns=turns, budget=4000, output_reserve=800,
)
print(f"kept {len(kept)} recent turns, dropped {len(dropped)} old ones, using ~{used} tokens")
print("dropped:", [d[0] for d in dropped])
# -> kept 9 recent turns, dropped 11 old ones, using ~3900 tokens
# Those 11 dropped turns are exactly what you'd SUMMARIZE (compression) so they're not lost.

Then you assemble the budgeted context — the dropped turns come back as a short running summary (the one piece that needs a model call):

# A budgeted window = system + a SUMMARY of old turns + RECENT turns verbatim
# + room reserved for the model's output. Each piece earns its slice.
def build_context(system, running_summary, recent_turns, user_msg):
    return "\n\n".join(filter(None, [
        f"## System\n{system}",
        f"## Summary so far\n{running_summary}",   # compression: old turns rolled up
        f"## Recent conversation\n{recent_turns}",  # selection: latest turns, verbatim
        f"## User\n{user_msg}",
    ]))
# The running_summary is produced by an LLM call once old turns are evicted:
#   running_summary = llm("Summarize these turns, keeping decisions & open questions:\n" + dropped_text)

That's the production shape: a few recent turns kept exact, the rest compressed into a summary, the system prompt always present, and headroom reserved for the reply. Frameworks (LangChain's summary-buffer memory, and Anthropic's auto-compaction) implement exactly this — but now you know what they're doing under the hood.

Pitfalls & Common Mistakes

Blind truncation. A sliding window that drops the turn where the goal/constraint was set quietly derails everything after. Protect key turns (or summarize, don't delete).
Over-aggressive summaries. Compress too hard and you lose the exact detail (an ID, a number, a decision) you needed. Keep recent turns verbatim; summarize only the old.
Forgetting the output reserve. If you fill the window to the brim, there's no room left for the model's answer. Always reserve output tokens.
Summarizing every turn. Compression costs a call — trigger it at a threshold (~70–80%), not on every message.
Dropping the system prompt to save space. Never. The rules are the highest-signal tokens in the window.
Confusing this with memory. Selection/compression manage what's in the window now; durable facts that should survive across sessions belong in memory — the next lesson.

🧪 Try It Yourself

The window-as-a-budget is best felt. In the widget below, keep adding turns and watch the bar fill: system prompt + growing history + the reserved output slice — until it overflows. That overflow is the moment a real chatbot errors out (or starts rotting).

Now reason it through: at the point of overflow, what would you keep verbatim, and what would you summarize? Sketch the budget — e.g., "system + last 5 turns exact + a 3-line summary of the rest + 600 tokens reserved for output." That allocation decision is context-window management in one move.

Window Manager — turn 1 sets the goal (plan a 3-day Tokyo trip on a $1000 budget). Grow the conversation and pick a strategy for fitting it into a finite context window. Keep everything and the request eventually overflows the window (a hard API error); use a sliding window of the last five turns and it fits but silently drops turn 1, so the model forgets the budget and city and wanders off-task; use the hybrid (roll the old turns into a summary, keep the recent ones verbatim, always keep the system prompt, reserve room for the output) and it fits AND preserves the goal. A live token bar shows each part’s slice and flags overflow, and a coherence indicator shows whether the goal survives — making concrete the two levers, selection (what to include) and compression (shrink what you keep).

Key Takeaways

LLMs are stateless: you re-send the whole history every call, so the window fills up and eventually overflows (or rots first).
You have two levers: selection (what to include — recency / sliding window or relevance-based) and compression (shrink it — rolling summarization, compaction, doc compression).
The production pattern is hybrid: recent turns verbatim + older turns summarized + system prompt always + an output reserve.
Tradeoffs: blind truncation kills coherence; over-summarizing loses detail; compression costs an extra call. Keep recent exact, summarize the old.
This manages the window now; durable cross-session facts belong in memory.

Next: Memory, State & History as Context — how agents remember across turns and sessions by writing facts outside the window and pulling them back when needed.