Language Models 101: Predicting the Next Token

Time to look under the hood. Large language models can write code, hold conversations, and reason through problems — so it's natural to imagine something elaborate inside. The reality is almost shockingly simple:

A language model does exactly one thing: predict the next token.

That's it. Chat, coding, translation, reasoning — all of it emerges from a model repeatedly guessing what comes next. We start the entire "how it works" section here because once this clicks, nothing later in the course will feel like magic.

You'll learn:

• The model's one job — and the autocomplete analogy that makes it obvious
• Why it outputs a probability distribution, not a single answer
• How whole paragraphs come from predict → append → repeat
• Where the apparent "understanding" actually comes from
• A tiny demo showing the model's real next-token probabilities

You already use a tiny language model every day: your phone's keyboard, suggesting the next word as you type. A large language model is the same idea — predict the next word — but trained on a huge fraction of everything ever written, so its guesses are extraordinarily good.

Give the model some text (the prompt) and its job is to answer one question: what token most likely comes next?

Input:   "The capital of France is"
Model:   → most likely next token: "Paris"

A quick word on "token": for now, think of it as roughly a word (or a chunk of one). Models don't work in whole words exactly — they work in tokens — but we'll make that precise in the very next lesson. For this lesson, token ≈ word is close enough.

Here's the subtlety that explains so much later. The model doesn't output the single word "Paris." It outputs a probability for every token in its vocabulary — tens of thousands of them — saying how likely each is to come next:

"The capital of France is" →
   Paris    91%
   the       3%   (e.g. "…the largest city…")
   a         2%
   located   1%
   Lyon      0.4%
   …         (every other token gets some tiny probability)

Then a separate step picks one token from this distribution (that's sampling — we devote a whole lesson to it in Section 5). Mechanically, the model produces a vector at the last position, a final "language head" layer turns it into one score per vocabulary token, and a softmax squashes those scores into probabilities that sum to 1.

This is the root of the probabilistic nature you met in the mindset lesson: because the model emits a distribution and we sample from it, the same prompt can yield different outputs — and a low-probability-but-wrong token is always possible.

If the model only predicts one token, how does it write whole paragraphs? It loops. This is called autoregressive generation:

1. Predict the next token from the prompt.
2. Append it to the text.
3. Feed the now-longer text back in and predict the next token.
4. Repeat until it predicts a special "stop" token (or hits your max_tokens).

"The capital of France is"            → "Paris"
"The capital of France is Paris"      → "."
"The capital of France is Paris."     → "It"
"The capital of France is Paris. It"  → "is"
…and so on, one token at a time.

Two consequences you already half-know:

• Generation is sequential — each token depends on all the ones before it, so longer outputs literally take more steps (and more time and cost — recall the token usage from the API lesson).
• The model re-reads the whole sequence every step, which connects directly to why context length and statelessness matter.

"It's just predicting the next word" is true — and yet the result can explain code or draft a legal summary. How?

Because predicting the next token well forces the model to learn everything underneath the text. To reliably continue:

• "2 + 2 =" → it must learn arithmetic.
• "The opposite of hot is" → it must learn word meaning.
• "def add(a, b): return" → it must learn code.
• The last line of a mystery novel → it must track the plot, characters, and logic.

Grammar, facts, reasoning, and style aren't programmed in — they're emergent side effects of getting very, very good at one simple objective across a massive fraction of human text. That's the big idea: a simple goal, at enormous scale, produces apparent intelligence. Everything else in this course is engineering on top of this phenomenon.

Where do the probabilities come from? From training that needs no human labels — the text labels itself:

1. Take a sentence from the training data.
2. Hide the next token; have the model predict it.
3. Compare its guess to the real token; nudge its internal weights to be a bit more right.
4. Repeat across trillions of tokens.

Because the "answer" is just the word that actually came next, you can do this on essentially all the text on the internet automatically. This is called self-supervised learning, and it's exactly why models could scale to where they are (recall lesson 1: "learned from the internet"). We'll go deeper on training in the Foundation Models section — for now, the takeaway is just: predict-the-next-token is both how it runs and how it learned.

This isn't a metaphor — you can inspect the actual probabilities. OpenAI exposes them via logprobs. Ask for a single next token and the model's top candidates with their probabilities:

math.exp(logprob) converts the log-probability back to a plain probability. There it is in black and white: the model considered the whole vocabulary, concentrated ~98% of its belief on "Paris," and left crumbs for everything else. Crank the prompt to something open-ended ("Write a story about") and you'll see the probability spread out across many plausible tokens — which is exactly when sampling choices start to matter.

Predict the next token. What word does a language model most likely predict after each prompt?

1. "The capital of France is" → ?
2. "The capital of France is not" → ?

→ 1: "Paris" (the most probable continuation). 2: not Paris — the single word not flips the whole distribution toward other cities. That's the entire model in a nutshell: it predicts the most likely next token given everything so far — nothing more, nothing less.

• "It's just autocomplete." Technically yes — but autocomplete trained on humanity's text at massive scale is a different beast. Don't let "just" fool you.
• It doesn't "look up" facts. It predicts plausible tokens. When the most plausible continuation isn't the true one, you get a confident-sounding wrong answer — the root of hallucination (and the reason RAG exists).
• It has no memory between calls. Each request predicts from only the text you send (recall API statelessness). "Memory" is something you engineer.
• A token isn't a word. We've used token ≈ word as a crutch; the next lesson makes it precise — and explains some genuinely weird model behavior along the way.

• A language model has one job: predict the next token — and outputs a probability distribution over the entire vocabulary, from which one token is sampled.
• Whole texts come from autoregressive generation: predict → append → repeat, one token at a time.
• Apparent understanding is emergent: getting great at next-token prediction across vast text forces the model to learn grammar, facts, and reasoning.
• It learns this via self-supervised training — the next word is the label — which is why it scaled.
• This single idea explains hallucination, statelessness, and why outputs vary — themes we'll build on throughout.

Next: we stop hand-waving the word "token" and make it precise — how text becomes numbers (BPE) — and you'll see why models can't reliably count the letters in "strawberry."