From Logits to Probabilities (Softmax)

We've built the model (pretraining) and given it manners (post-training). Now we open the hood on the part you actually control every time you call an API: how a model generates text, one token at a time.

Here's the fact that surprises almost everyone: a language model never outputs a word. At each step it outputs a number for every single token in its vocabulary — a score saying how likely each one is to come next. This lesson is about that exact moment: how those raw scores (logits) get turned into clean probabilities by a function called softmax.

Why start here? Because every generation setting you'll ever touch — temperature, top-p, top-k, logprobs, even why models hallucinate — operates on this one step. Understand logits → softmax and the rest of this section (and a lot of mysterious model behavior) clicks into place.

You'll learn:

• What logits are, and why they aren't yet probabilities
• How softmax converts them into a real probability distribution
• Why softmax exponentiates (the clever part)
• The properties that make this the foundation of all sampling
• A first peek at the temperature knob (full treatment soon)

From the tokenization and transformer lessons, recall the core loop. A model generates autoregressively — one token at a time, each new token appended and fed back in to predict the next:

"The cat sat on the"  →  [model]  →  "mat"
"The cat sat on the mat"  →  [model]  →  "."
... and so on.

We said the model "predicts the next token." But what does the transformer's final layer actually emit to make that prediction? Not the word "mat." It emits a long list of raw scores — one per possible token — and that list is where every generation control begins.

After the input flows through all the transformer layers, the final hidden vector is projected (by the output / 'unembedding' layer) into a vector with one entry for every token in the vocabulary. Modern vocabularies are huge — typically ~100,000 to 200,000 tokens — so this is a list of ~100k+ numbers, produced every single step.

Each number is a logit: a raw, unnormalized score for how strongly the model favors that token as the next one. Crucially, logits are not probabilities:

• they can be negative (e.g. −1.5) or large positive,
• they don't fall between 0 and 1,
• they don't sum to anything meaningful.

Think of logits as the model's raw "enthusiasm scores." Higher means more favored, but you can't yet read them as percentages. Here's a tiny toy example (real vocab = 100k+, shown with 5 tokens for "The cat sat on the ___"):

  token     logit
  -----     -----
  mat        3.0
  floor      2.1
  couch      1.2
  roof       0.0
  banana    -1.5

We can see mat is favored — but "3.0" isn't "how likely." We need to convert.

Softmax is the function that converts a vector of logits into a proper probability distribution. It does two simple things:

1. Exponentiate every logit: compute e^(logit). This makes every value positive (even the negative logits) and stretches the gaps between them.
2. Normalize: divide each exponentiated value by the sum of all of them, so the results add up to exactly 1 (100%).

In one formula:

                exp(logit_i)
  P(token_i) = ----------------------
               Σ_j exp(logit_j)

Run our toy logits through it and the raw scores become readable probabilities:

Now it reads cleanly: the model thinks mat is the next token about 61% of the time, floor 25%, and banana a negligible 1% — and those five add up to 100%. That's a probability distribution we can actually pick from.

Why not just divide each logit by the total and skip the e^x? Two reasons make exponentiation the right tool:

1. It handles negatives gracefully. Logits can be negative; you can't have a negative probability. Exponentiating maps any real number — negative or positive — to a positive value (e^(-1.5) ≈ 0.22, still > 0). Plain division would break on negative scores.
2. It amplifies the lead. e^x grows fast, so a token with a somewhat higher logit ends up with a much higher probability. A logit gap of 0.9 between mat (3.0) and floor (2.1) becomes a 61%-vs-25% gap — the model expresses confidence, not a flat tie. The bigger the logit gaps, the sharper (more decisive) the distribution.

And it preserves order: higher logit always → higher probability (softmax is monotonic). It never reshuffles the model's ranking — it just rescales raw scores into a confident, well-formed distribution.

Four facts about a softmax distribution — worth locking in, because the rest of the section leans on them:

• Every probability is between 0 and 1. Readable as a percentage.
• They sum to exactly 1 (100%). It's a real distribution over the whole vocabulary — every possible next token gets some share, even if tiny.
• Ranking is preserved. The highest-logit token is always the highest-probability token.
• Only the gaps matter, not absolute values. Adding the same constant to every logit changes nothing; what shapes the distribution is how far apart the logits are. Big gaps → a peaky, confident distribution; small gaps → a flat, uncertain one.

That last point is the seed of the next idea: if the spread of the logits controls how confident the distribution is, then a knob that stretches or squashes that spread would let us dial the model's confidence up or down…

…and that knob is temperature. Before applying softmax, the model can divide every logit by a number T:

• Low temperature (e.g. 0.2) stretches the gaps → a sharper, more confident distribution (it almost always picks mat).
• High temperature (e.g. 1.5) squashes the gaps → a flatter, more random distribution (floor, couch, even roof get real chances).

That's the entire mechanism behind "make the model more creative" vs. "make it more focused" — it's just reshaping this softmax distribution. We'll give temperature, top-k, and top-p their own full lesson shortly. For now, the key takeaway: every model, on every call, ends each step by producing a softmax probability distribution — and the sampling controls all act right here.

This isn't abstract math — it's the control panel for everything you'll do at inference:

• Every sampling parameter acts on this step. Temperature, top-p, top-k, and logit_bias all reshape or trim the logits/softmax before a token is picked. You can't reason about them without this picture.
• logprobs are literally these probabilities (logged). When an API returns logprobs, it's handing you log(P) for the chosen tokens — a direct read on the model's confidence. High logprob = the model was sure; low = it was guessing. Hugely useful for evaluation, RAG confidence, and classification.
• It demystifies model behavior. Why outputs vary run-to-run, why "temperature 0" is near-deterministic, and where hallucinations leak in — all trace back to this distribution (the next lessons build directly on it).
• It's the foundation of the whole section. Greedy vs. sampling, temperature/top-k/top-p, determinism, hallucinations — every one of them is a different way of dealing with the softmax distribution you just learned to read.

Here are the model's raw scores (logits) for “The cat sat on the ___”. Watch them become a clean probability distribution via softmax. Drag temperature to feel the key idea: only the gaps between logits matter — a small temperature stretches the gaps (peaky, confident); a large one squashes them (flat). (Temperature is your teaser here — it gets its own lesson next.)

• "The model outputs a word." No — it outputs a logit for every token in the vocabulary (~100k+ numbers), every step. The word is chosen afterward.
• "Logits are probabilities." No — they're raw scores: can be negative, unbounded, don't sum to 1. Softmax turns them into probabilities.
• "Softmax picks the token." No — softmax only produces the distribution. Choosing a token from it (greedy or sampling) is a separate step — the next lessons.
• "Bigger logits everywhere = different result." No — only the gaps between logits matter; adding a constant to all of them leaves the probabilities unchanged.
• "Temperature changes the model." No — it just rescales the logits before softmax. Same model, reshaped distribution.

• At each step a model emits logits — one raw score per vocabulary token (~100k+) — not a word and not probabilities.
• Softmax converts logits into a true probability distribution: exponentiate (all positive), then normalize (sum to 1) — P_i = exp(z_i) / Σ exp(z_j).
• Exponentiating handles negative scores and amplifies the lead, turning raw scores into a confident, ranking-preserving distribution.
• Only the gaps between logits shape the distribution — which is exactly what temperature stretches or squashes.
• This logits → softmax step is the foundation of all generation controls (temperature, top-p/k, logprobs) and the rest of this section.

Next: now that we have a probability distribution over the next token, the obvious question is how do we pick one? We start with the two basic strategies — greedy decoding vs. sampling — and why that single choice changes everything about the output.