Temperature, Top-k & Top-p Explained

Last lesson left us with a problem: sampling gives a model its creativity, but unconstrained it can occasionally grab a junk token off the long tail and derail. These three knobs — temperature, top-k, and top-p — are how you keep the creativity and cut the junk. They are, without exaggeration, the most-used controls in all of AI engineering: they're the parameters in nearly every API call you'll ever write.

The good news is they're far simpler than their reputation suggests once you have last two lessons in hand. Every one of them is just a small operation on the softmax distribution (Lesson 1) performed before the sampling pick (Lesson 2).

You'll learn:

• The mental model: reshape vs trim
• Temperature — sharpen or flatten the whole distribution
• Top-k — keep a fixed number of candidates
• Top-p (nucleus) — keep an adaptive number of candidates (and min-p, the modern alternative)
• How they combine, the #1 mistake, and what values to use per task

Before the details, hold this framing — it makes all three knobs obvious. They do two different jobs on the distribution:

• Temperature reshapes it — makes the whole distribution peakier or flatter (controls how confident the model acts).
• Top-k / top-p trim it — cut off the tail of unlikely tokens so the dice can only land on reasonable options (controls how many candidates are even allowed).

They're often used together, and the order is fixed: reshape first (temperature), then trim (top-k/p), then sample. Keep "reshape vs trim" in mind and you'll never confuse them again.

Temperature (T) is a single number that divides every logit before softmax (softmax(logit / T)). Recall from Lesson 1 that only the gaps between logits shape the distribution — temperature stretches or squashes those gaps:

• Low temperature (e.g. 0.2) — divides by a small number → bigger gaps → a peakier, more confident distribution. The top token dominates; output is focused and near-deterministic. (T → 0 ≈ greedy.)
• High temperature (e.g. 1.5) — smaller gaps → a flatter distribution. Unlikely tokens get real chances; output is diverse, surprising, sometimes wild.
• T = 1 leaves the model's natural distribution unchanged.

It does not add or remove tokens — every token still has some probability. It only changes how confident the model is. Watch the same five-token distribution reshape as temperature rises:

Top-k trims the distribution by a simple rule: keep only the k highest-probability tokens, throw away the rest, renormalize the survivors to sum to 1, then sample.

With k = 2 on our distribution, only mat and floor survive; couch, roof, and banana are zeroed out — so the model cannot derail onto banana, no matter how unlucky the dice. Typical values are 40–100.

Top-k's limitation is that k is a fixed count that ignores the shape of the distribution:

• When the model is very confident (one token at 95%), k = 40 still keeps 39 low-quality alternatives.
• When the model is genuinely uncertain (50 roughly-equal good tokens), k = 40 may chop off perfectly good options.

A fixed cutoff is blunt. We'd prefer one that adapts to how confident the model is — which is exactly top-p.

Top-p (also called nucleus sampling) trims by cumulative probability instead of by count: keep the smallest set of top tokens whose probabilities add up to at least p, drop the rest, renormalize, and sample.

With p = 0.9 on our distribution: mat (61%) + floor (25%) = 86% — not yet 90%, so add couch (10%) → 96% ≥ 90%. The nucleus is {mat, floor, couch}; roof and banana are cut. Typical values: 0.9–0.95.

The magic is that the cutoff adapts to the model's confidence:

• Confident step (one token at 95%) → the nucleus is tiny (maybe just 1–2 tokens) → it stays focused.
• Uncertain step (many plausible tokens) → the nucleus grows to include them → it stays creative.

That adaptivity is why top-p is the mainstream default on most chat APIs, generally preferred over top-k.

Worth knowing because it's now common: min-p keeps any token whose probability is at least a fraction of the top token's probability — e.g. min_p = 0.1 keeps everything ≥ 10% as likely as the best token, and cuts the rest.

Its advantage is robustness: it stays anchored to the model's current confidence, so it doesn't blow up the candidate set at high temperature the way top-p can. By 2026 min-p is the default in much of the open-source stack (llama.cpp, vLLM, Ollama, Transformers), with typical values 0.05–0.1. If you're running local/open models, prefer min-p; on the big commercial APIs, top-p + temperature is still the standard combo.

When you set several at once, they apply in a fixed order: temperature reshapes the logits first → top-k/top-p trim the result → then the model samples from what's left. (This is why high temperature with top-p behaves differently from high temperature alone — the reshape happens before the trim.)

The #1 mistake: tuning temperature and top-p at the same time. They both control "how adventurous" the output is, so moving both at once makes the effect confusing and hard to reproduce. Provider guidance (OpenAI's included) is explicit: change one or the other, not both. Pick temperature as your main creativity dial or pick top-p — and leave the other at its default.

You rarely need to invent values — start from these field-tested defaults and adjust:

Rule of thumb: lower = safer/more deterministic, higher = more creative/varied. When something feels too repetitive or robotic, raise temperature; when it feels too random or off-topic, lower it (or tighten top-p).

These knobs are your day-to-day control panel:

• They're in every API call. Knowing them turns "the model is too random / too bland" from a mystery into a one-line fix.
• temperature = 0 for anything you must trust or reproduce — extraction, JSON, classification, evals, tests, caching. Don't sample when there's one right answer.
• Crank it up for ideation — if brainstorms feel samey, raise temperature; that flatter distribution is literally where novelty comes from.
• Don't double-tune. Move temperature or top-p. Log whatever you set, because it directly affects reproducibility.
• Match the runtime: top-p on commercial APIs, min-p on local/open stacks. Same idea (trim the tail), different default knob.

These controls shape which token gets chosen. The next lesson covers a different family of controls — ones that decide when to stop, how long to go, and how to discourage repetition (stop sequences, max tokens, penalties).

Reading about temperature and top-p is one thing — feel them. Drag the knobs below and watch the same logits become a different distribution in real time:

• Pull temperature toward 0 → the bars sharpen until mat dominates (near-greedy). Push it up → they flatten and the long tail wakes up.
• Pull top-p down → watch the tail tokens get trimmed (grayed out) and the survivors renormalize.
• Pull top-k down → only a fixed number of top tokens survive — a blunt cut, vs top-p's adaptive one (compare how each responds when the distribution is peaky vs flat).
• Hit 🎲 Roll the dice at low vs high temperature → see the pick go from boringly-consistent to wildly-varied.

This is the whole lesson in your hands: temperature reshapes, top-p trims, then you sample.

• "Temperature adds or removes words." No — it reshapes the whole distribution (peaky vs flat). Top-k/top-p are what remove tokens (trim the tail).
• "Top-k and top-p are the same." No — top-k keeps a fixed count; top-p keeps an adaptive set by cumulative probability. Top-p adjusts to the model's confidence.
• "Higher temperature = smarter." No — it's just more random. Too high → incoherent. Creativity ≠ correctness.
• "Set temperature and top-p for fine control." No — tune one; moving both is the classic mistake.
• "Temperature 0 is perfectly deterministic." Nearly — it's effectively greedy, but other factors can still cause tiny variation (next-next lesson).

• All three knobs operate on the softmax distribution before the sampling pick. Mental model: temperature reshapes; top-k/top-p trim.
• Temperature (softmax(logit/T)): low → peaky/focused (≈greedy at 0); high → flat/creative. It changes how confident, not which tokens exist.
• Top-k keeps a fixed number of top tokens; top-p (nucleus) keeps an adaptive set summing to ≥ p — top-p is the mainstream default; min-p is the modern open-source choice.
• Order: temperature → trim → sample. Don't tune temperature and top-p together.
• Defaults: ~temp 0 for factual/structured, ~0.7 / top-p 0.9 for chat, ~1.0+ for creative.

Next: the other generation controls — stop sequences, max tokens, and repetition/frequency penalties — which govern when output ends and how to keep it from looping.