Tokens, Context Windows & Why They Matter

Every model — local or cloud, big or small — has one hard, unavoidable limit: how much text it can consider at once. That limit is the context window, and it's measured in tokens.

This single concept quietly governs almost everything you'll build: how much you can feed a model, why chatbots eventually "forget" the start of a conversation, what your requests cost, and why you sometimes hit a wall. We met tokens back in Section 2 (text → tokens via BPE); now we use them as the practical currency and budget of every API call.

You'll learn:

• A quick refresher on tokens (and a rule of thumb for estimating them)
• What the context window is — the model's entire working memory
• Why models are stateless and "forget" (and what that costs)
• How big windows are in 2026 — and what happens when you exceed one
• Why this shapes cost, memory, and design (previewing the next lessons)

From Section 2: models don't read characters or words — they read tokens, sub-word chunks produced by the tokenizer (BPE). "unbelievable" might be un + believ + able; common words are a single token, rare ones split into several.

You don't need to count by hand — just keep a rule of thumb for English:

• ~4 characters ≈ 1 token
• ~750 words ≈ 1,000 tokens (≈ 1.3 tokens per word)

The two facts that matter for this lesson: everything is measured in tokens, and both the text you send (input) and the text the model generates (output) count. For exact numbers, use a tokenizer (e.g. OpenAI's tiktoken, or the providers' token-counting endpoints) — but the rule of thumb is enough to reason about budgets.

The context window is the maximum number of tokens a model can hold in mind for a single request — its entire working memory. And here's the crucial part: everything has to fit inside it, together:

• the system prompt (instructions/persona),
• the entire conversation history so far,
• your new message (input),
• room for the model's response (output),
• plus, when relevant, reasoning tokens (for reasoning models) and tool definitions/results.

Think of it as one fixed-size desk. Books (system prompt, history, your question) and blank paper for the answer all have to share the same desk. Add too much and something falls off the edge. As a conversation grows, the history takes up more and more of that desk:

Recall from the API-anatomy lesson: an LLM call is a stateless function. The model keeps no memory between calls — each request is judged entirely on what's in that request's context window.

So how does a chatbot "remember" what you said three turns ago? It doesn't — your app re-sends the entire conversation history every single turn. The "memory" you experience is just the transcript being replayed into the context window each time.

This has two big consequences:

1. Long conversations cost more and more. Every turn re-sends a growing history, so token usage (and cost, and latency) climbs as the chat goes on.
2. Eventually it overflows. Once the history + new input + output would exceed the window, something must give — the oldest turns get dropped, or the call errors. That's why assistants "forget" the beginning of very long chats.

Windows have grown enormously. As of 2026:

• ~1 million tokens is now common on frontier models — Claude Opus 4.8, GPT-5.5, Gemini 3.1, and others.
• Even larger exist: Gemini 3.1 reaches 2M, and Llama 4 Scout leads at ~10M tokens.
• Output is capped separately and is much smaller — often around 128K tokens max, regardless of the input window.
• Local/open models you run yourself are usually smaller (commonly 8K–128K), bounded by your hardware.

For scale: 1M tokens ≈ 750,000 words ≈ 6–8 full novels — you can drop an entire codebase or a stack of documents into a single prompt. Just remember the input window and the output limit are two different numbers: a huge input window doesn't mean a huge reply.

Push past the window and one of two things happens: the API returns an error, or your framework silently truncates — usually dropping the oldest messages. Anything outside the window simply does not exist for the model; it can't reason about text it can't see.

One honest caveat worth planting now: the advertised window is a capacity, not a promise of perfect recall. In practice, models attend less reliably to the middle of very long contexts — effective recall degrades well before the stated maximum, so a 1M-token window doesn't mean 1M tokens of flawless attention. (We give long-context limits a full treatment in a later lesson — for now, just don't assume "it fits" means "it's fully used.")

The context window is a constraint you'll design around constantly:

• It bounds what you can feed. Whole documents, long chats, big codebases — does it fit? Tokens are how you check.
• It drives cost and latency. More tokens in the window = more to pay for and more to process. (The exact token math is the very next lesson.)
• Long chats need management. Because history is re-sent and grows, real apps trim or summarize old turns to stay within budget — the heart of context engineering (a later container).
• Always leave room for the output. Input and output share the window; if you fill it with input, there's nothing left for the answer.
• Count before you send. Use a tokenizer to estimate tokens for big inputs, so you don't get surprised by truncation or errors in production.

The context window is a fixed budget shared by the system prompt, the whole conversation history, your new input, and room for the reply. Hit + Add a turn a bunch of times and watch the History bar grow — because the model is stateless, every turn re-sends the entire transcript. Keep going and it overflows: the oldest turns must be dropped (the bot 'forgets' the start) or the call errors. This is why long chats get expensive and forgetful.

• "The model remembers our conversation." No — it's stateless; your app re-sends the whole history each call. The window is its only memory.
• "Only my input counts toward the window." No — system prompt + history + input + output (and reasoning/tools) all share it.
• "A 1M-token window means a 1M-token answer." No — output has its own, much smaller cap (often ~128K).
• "Tokens are words." Roughly, but not exactly — ~0.75 words per token; rare words split into several tokens.
• "A bigger window means perfect recall." No — effective recall degrades in very long contexts; capacity ≠ flawless attention.

• Everything is measured in tokens (~4 chars ≈ 1 token; ~750 words ≈ 1,000 tokens); input and output both count.
• The context window is the model's entire working memory — system + history + input + output must all fit in it together.
• Models are stateless: chat "memory" is the whole history re-sent every turn, so long conversations grow in cost and eventually overflow (oldest turns dropped, or an error).
• 2026 windows: ~1M common (Claude Opus 4.8, GPT-5.5, Gemini 3.1), up to 2M–10M; output is capped separately (~128K); local models smaller.
• Capacity ≠ recall — big windows attend less reliably to the middle (deep dive later).

Next: now that you think in tokens, we make it concrete and financial — API pricing and token math: exactly how those input and output tokens turn into dollars, and how to estimate (and cut) the cost of a feature.