Controlling Output: Stop Sequences, Max Tokens, Penalties

The last two lessons controlled which token gets chosen. This lesson controls two different things that matter just as much in real systems: when the model stops, and how to keep it from repeating itself.

These controls are less glamorous than temperature — but they're where a huge share of real production bugs live: responses that run on forever (and rack up cost), models that loop ("...and that's great, and that's great, and..."), JSON that never closes, and assistants that helpfully continue the conversation as the user. The right parameter turns each of these from a mystery into a one-line fix.

You'll learn:

• How generation ends naturally (the EOS token)
• Max tokens — a hard length cap (and the max_completion_tokens twist for reasoning models)
• Stop sequences — halting at a marker you choose
• Frequency & presence penalties — two different ways to fight repetition
• logit_bias — surgically banning or forcing specific tokens

Group these controls by the job they do and they stop feeling like a random grab-bag:

1. When to stop — the EOS token (natural end), max tokens (a length cap), and stop sequences (halt at a marker). These decide where generation ends.
2. How not to repeat — frequency and presence penalties, plus logit_bias. These nudge the logits before sampling (same spot temperature and top-p act, from last lesson) to discourage — or surgically control — specific tokens.

Keep those two buckets in mind; everything below slots into one of them.

Recall the autoregressive loop: the model generates one token, appends it, and predicts the next — forever, unless something tells it to stop. So how does a normal response end?

The vocabulary includes a special end-of-sequence (EOS) token. During post-training the model learned to emit EOS when a response feels complete. Each step, EOS is just another candidate in the softmax distribution; the moment the model samples the EOS token, generation halts. That's the "natural" stop behind every well-formed answer.

But you can't always rely on it alone — the model might ramble past where you want, never emit EOS in a loop, or need to stop at a structural point (like the end of a JSON object). That's what the next controls are for.

Max tokens sets the maximum number of tokens the model may generate. Hit the cap and generation stops immediately — even mid-word. It's your budget and safety belt against runaway output.

Two things people get wrong:

• It is not a "be concise" instruction. It doesn't make the model plan a shorter answer — it just truncates when the limit is reached, often leaving a sentence (or your JSON) chopped off. To get genuinely shorter responses, ask for them in the prompt; use max tokens to bound cost and prevent loops.
• max_tokens vs max_completion_tokens. Newer APIs use max_completion_tokens, which counts both the visible output and the hidden reasoning tokens of reasoning models. So a reasoning model needs a much larger budget — set it too low and the model can burn the whole allowance "thinking" and return nothing.

Also don't confuse this with the context window — that's the total input + output the model can hold; max tokens caps only the output portion.

A stop sequence is a string that, the instant the model generates it, halts generation — and the stop string itself is removed from the returned text. You can usually supply up to ~4. This is precise, content-based stopping, in contrast to max tokens' blunt length cap.

Classic uses:

• End a code block: stop on ``` so the model can't keep typing after the snippet.
• Prevent role-play leakage: stop on "\nUser:" (or "\nHuman:") so the model answers its turn and doesn't hallucinate the user's next message — a very common bug in hand-rolled chat loops.
• Bound structured output: stop on a delimiter you define (e.g. "END" or "</answer>") to cut cleanly at the end of the part you care about.

prompt: "List three fruits, then stop."
stop:   ["4."]
→  "1. Apple  2. Banana  3. Cherry  "   (halts before ever writing "4.")

When output gets repetitive, two penalties help — and they work differently. Both subtract from the logits of tokens already used (before sampling), but they count usage differently:

• Frequency penalty scales with how often a token has appeared. Use the word "great" five times and it gets pushed down hard. → Best for killing verbatim repetition and loops.
• Presence penalty is a flat penalty applied if a token has appeared at all (even once). → Best for encouraging new topics and vocabulary — nudging the model to move on rather than circle the same words.

Both typically range about −2.0 to +2.0; positive values discourage repetition. Useful settings are gentle — 0.1 to 0.8. Crank them too high and the model starts avoiding necessary words and reads strangely. (frequency_penalty, presence_penalty, and logit_bias are OpenAI parameters — Anthropic's API doesn't expose them; on Claude, curb repetition via the prompt and stop sequences.) Here's the full control panel for this lesson:

When penalties are too broad, logit_bias gives you a scalpel. You map specific token IDs to a bias value (about −100 to +100) that's added to those tokens' logits before sampling (right where last lesson's knobs operate):

• −100 ≈ effectively ban a token (it'll never be chosen).
• A positive bias makes a token more likely — handy to force a format or a yes/no vocabulary.

Example: ban a profanity, or force a classifier to choose only between the tokens for "yes" and "no". It's powerful but finicky (you work in token IDs, and tokenization can be surprising), so reach for it only when prompt wording and penalties aren't enough.

How these combine in practice:

• Strict JSON extraction: temperature 0, a sensible max_tokens cap, and a stop sequence if you're streaming a known boundary. Deterministic and bounded.
• Hand-built chat loop: add stop: ["\nUser:"] so the model never writes the user's next turn, and always set max_tokens so a runaway reply can't drain your budget.
• Fixing a repetition loop: bump frequency penalty to ~0.3–0.6. If it keeps hammering the same topic, add a little presence penalty.
• Reasoning model returns empty: your max_completion_tokens is too low — it spent the budget thinking. Raise it.
• Banning a word / forcing a choice: logit_bias (−100 to ban, positive to force).

This is the difference between a demo and a robust system:

• Always set max_tokens in production. It's your circuit breaker against infinite loops and surprise bills — non-negotiable for anything user-facing or automated.
• Stop sequences keep structure clean. They're the simplest fix for "the model kept going past my code block" or "it answered as the user."
• Penalties are your repetition first-aid. See looping or samey text? Reach for frequency (verbatim) or presence (topics) before re-engineering the prompt.
• max_completion_tokens ≠ max_tokens for reasoning models — budget for the hidden thinking, or you'll get empty responses.
• These pair with last lesson's knobs. Penalties and logit_bias live on the logits; stop/max live on the stopping rule. Together they fully specify what comes out and when it ends.

Pick the control. Match each production bug to the fix (max_tokens · stop sequence · frequency_penalty):

1. The model keeps typing past your code block. → ? 2. In your chat loop, it hallucinates the user's next message. → ? 3. It loops: 'great, and great, and great…' → ?

→ 1 & 2: a stop sequence (``` and \nUser:). 3: frequency_penalty (~0.3–0.6). And always set max_tokens as your runaway-cost circuit breaker.

• "Max tokens makes the model concise." No — it truncates at the limit (possibly mid-sentence). For brevity, ask in the prompt; use max tokens to bound cost.
• "max_tokens and max_completion_tokens are the same." For reasoning models, no — max_completion_tokens also counts reasoning tokens, so set it higher.
• "The stop sequence appears in the output." No — it's the trigger to halt and is removed from the returned text.
• "Frequency and presence penalties are interchangeable." No — frequency scales with count (kills verbatim loops); presence is flat (encourages new topics).
• "Penalties change which token is most likely arbitrarily." No — they subtract from logits of already-used tokens before sampling; everything else is unchanged.

• Two jobs: when to stop (EOS token, max tokens, stop sequences) and how not to repeat (frequency/presence penalties, logit_bias).
• EOS is the natural end; max tokens is a hard length cap (a budget/safety belt that truncates, not a conciseness setting) — use max_completion_tokens for reasoning models since it counts thinking tokens.
• Stop sequences halt at a marker you choose (e.g. ``` or \nUser:) and are excluded from the output.
• Frequency penalty fights verbatim repetition (scales with count); presence penalty encourages new topics (flat). Keep both gentle (~0.1–0.8).
• logit_bias surgically bans (−100) or forces specific tokens by adding to their logits.

Next: we've now seen randomness and all the knobs that shape it — so the natural question is how 'random' is a model, really? We dig into the probabilistic nature of AI: determinism, seeds, and why even temperature = 0 isn't a perfect guarantee.