Greedy Decoding vs Sampling

Last lesson ended with a clean probability distribution over the next token: for "The cat sat on the ___", the model said mat 61%, floor 25%, couch 10%, and so on. Now comes the question that decides everything about how the model feels: given that distribution, how do we actually pick one token?

There are two fundamental strategies, and the gap between them is enormous:

• Greedy decoding — always take the single most likely token.
• Sampling — roll the dice, choosing tokens in proportion to their probabilities.

This one choice is why the same model can act like a precise calculator (give it the same input, get the same answer every time) or like a creative writer (a fresh, surprising response on every call). Getting it right is one of the most practical skills in AI engineering.

You'll learn:

• How greedy decoding works, and its repetition trap
• How sampling works, and how it can go off the rails
• Where beam search fits in
• A clear decision guide for which to use when

Keep last lesson's picture in mind — it's the only thing both strategies start from:

  mat     61%
  floor   25%
  couch   10%
  roof     3%
  banana   1%

The model has already done its work; this distribution is fixed. Greedy and sampling don't change the probabilities — they only change the rule for picking a token from them. That's the whole lesson in one sentence. Let's see each rule.

Greedy decoding is the simplest possible rule: at every step, pick the token with the highest probability — the argmax — and ignore the rest.

For our example, greedy picks mat, 100% of the time. Run it again with the same prompt and you get mat again. This makes greedy:

• Deterministic — same input → identical output, every time.
• Fast and cheap — no randomness, no extra bookkeeping, just "take the max."

That predictability is exactly what you want for tasks with a single right answer: pulling a number out of a document, classifying a ticket, emitting strict JSON, or calling a tool. You don't want creativity in a date field.

Greedy has a famous weakness: on longer, open-ended text it becomes repetitive and bland, and can fall into outright loops ("I think that I think that I think that…"). This is the well-documented neural text degeneration problem.

The deeper reason is subtle and worth internalizing: always picking the locally most-likely token does not produce the most-likely (or best) overall sentence. Greedy has tunnel vision — it grabs the best next word and can paint itself into a corner, missing a better sentence that started with a slightly less likely word. Human language is full of moments where the single highest-probability next token is a safe, dull choice; relentlessly taking it yields safe, dull, often repetitive prose.

So greedy is excellent for short, factual outputs and a poor fit for anything that should feel natural or varied.

Sampling keeps the model's uncertainty alive. Instead of always taking the top token, it picks randomly — but weighted by the probabilities. Think of a spinner where each token gets a slice of the wheel sized to its probability:

• mat is chosen ~61% of the time,
• floor ~25%,
• couch ~10%,
• roof and banana rarely, but not never.

Because there's real randomness, the output varies run to run — the same prompt can yield different (often equally good) responses. That's the source of a model's creativity, variety, and natural feel: it's not locked into the single safest word, so it can produce fresh phrasings and ideas. This is why chat and writing assistants sample by default.

Sampling's freedom is also its risk. Because every token has a non-zero chance, pure (unconstrained) sampling will occasionally pick a genuinely bad, low-probability token — say banana — and once a weird word is on the page, the model continues from it, and the response can derail into incoherence.

So we want the best of both: keep the creative variety of sampling, but cut off the long tail of junk tokens so the dice can only land on reasonable options. That is exactly what the next lesson is about — temperature (how sharp or flat to make the distribution) and top-k / top-p (how much of the tail to keep). For now, just hold the trade-off: greedy is safe but dull; raw sampling is lively but risky; the knobs let you tune between them.

There's a classic middle path worth knowing: beam search — think of it as greedy with look-ahead. Instead of committing to one token at each step, it keeps the top B candidate sequences ("beams") alive at once, extends each, and prunes back to the best B — trying to maximize the probability of the whole sequence, not just the next token. This sidesteps greedy's tunnel vision.

Beam search is deterministic and shines on tasks with a single correct answer — classically machine translation and some structured generation. But it's memory-hungry (you track B sequences at once, which limits how many users fit on a GPU) and, for open-ended chat, it tends to produce safe, repetitive text — so modern conversational LLMs rarely use it. Know it exists; reach for greedy or sampling by default.

The decision comes down to: does this task have one right answer, or many good ones?

A crucial connector: in practice, "greedy" is what you get at temperature ≈ 0. Most APIs don't expose a literal "greedy" switch — you set temperature=0 (or near it) for deterministic/factual behavior and raise it for creative behavior. So in real code, the greedy-vs-sampling choice usually shows up as the temperature you pick — which is exactly the dial we open up next.

This is the single biggest lever on the character of your output:

• Reproducibility: evals, tests, caching, and debugging need stable outputs — use greedy / temp 0 so the same input gives the same result.
• "Why did I get a different answer this time?" — almost always because you were sampling. That's a feature for chat, a bug for a data pipeline.
• Structured/JSON/tool use: lean deterministic — you want the correct token, not a creative one.
• Quality vs. determinism isn't free: greedy can be repetitive on long text; if you need both coherence and determinism, that's a prompt/structure problem, not just a decoding one.
• It frames the next lesson: once you've decided you want sampling, temperature/top-k/top-p are how you make it smart instead of chaotic.

Same distribution, two selection rules — try both. Hit 🎯 Greedy (argmax) a few times: it always picks mat — deterministic and a little boring. Now hit 🎲 Sample repeatedly: it picks weighted by probability, so the answer varies (mostly mat, sometimes floor…). Crank temperature up and sample again to watch it get wilder. That single choice — argmax vs weighted dice — is the whole lesson.

• "Greedy gives the best answer." It gives the locally-best token each step — not the best overall sentence. Tunnel vision causes repetition.
• "Sampling means random/incoherent." It's weighted random — likely tokens still dominate. Incoherence only creeps in from the low-probability tail (which top-k/top-p trims).
• "Greedy and sampling change the model's probabilities." No — the softmax distribution is fixed; they only change the selection rule.
• "Beam search is best for chat." No — it's for single-answer tasks (e.g. translation); it's costly and bland for open-ended generation.
• "There's a 'greedy' button." Usually not — in practice you get greedy via temperature ≈ 0.

• Both strategies start from the same softmax distribution and differ only in the rule for picking a token.
• Greedy (argmax): always take the top token → deterministic, fast, great for factual/structured tasks, but repetitive/bland on long open-ended text (locally best ≠ globally best).
• Sampling: pick weighted by probability → varied, creative, natural, non-deterministic, but risks derailing on low-probability tokens.
• Beam search is a deterministic look-ahead option for single-right-answer tasks (e.g. translation); rarely used for chat.
• In practice the choice surfaces as temperature: ~0 ≈ greedy/deterministic; higher = sampling/creative.

Next: we open the controls that make sampling usable — temperature, top-k, and top-p — the knobs that decide how bold or how safe the dice roll is.