Where Hallucinations Come From

This whole section built one idea: a language model is a probabilistic next-token engine — it turns context into a distribution over plausible next tokens (softmax) and samples one. Now we meet the most famous, most consequential side effect of that design: hallucination — when a model produces fluent, confident text that is simply false.

Here's the reframe that makes everything click: a hallucination isn't a weird glitch or a bug to be patched away — it's the expected behavior of a system optimized for plausibility rather than truth. Once you see why it's baked into how these models work, you'll stop being surprised by it and start designing around it — which is what every serious AI engineer does.

You'll learn:

• What a hallucination actually is (intrinsic vs extrinsic)
• The root cause: plausibility ≠ truth (and no truth-check anywhere)
• Why models bluff instead of saying "I don't know" (the 2025 incentive finding)
• The situations that trigger hallucinations most
• How they're reduced (a preview of grounding/RAG, covered in depth later)

A hallucination is output that is fluent and confident but false or unsupported — a made-up fact, a fake citation, an invented API method, a wrong date stated with total assurance.

Crucially, the model isn't lying — lying requires knowing the truth and choosing to deceive. The model has no intent and no notion of truth; it's producing what looks like a good continuation. That's why hallucinations are so dangerous: they arrive in the same confident, well-formed prose as correct answers.

Two flavors are worth naming:

• Intrinsic hallucination — the output contradicts a source you gave it (e.g. you paste a contract and it misstates a clause).
• Extrinsic hallucination — the output is fabricated and ungrounded, not supported by any source (e.g. a citation to a paper that doesn't exist, a plausible-but-fake statistic).

Here is the deepest reason, and it's the payoff of this entire section. At every step the model picks the most plausible next token — the one that best fits the patterns in its training data. Nowhere in the pipeline — from training data, to learned weights, to next-token prediction — is the output ever checked against reality. There is no fact-checker, no truth gate.

The consequence is unsettling: a fluent falsehood and a fact can look equally plausible token-by-token. "The Eiffel Tower opened in 1889" (true) and "The Eiffel Tower opened in 1856" (false) are both grammatical, confident, ordinary-sounding sentences. The model has a strong sense of which continuation is likely; it has no separate sense of which is true.

Compounding the problem: generation always emits a token. There's no built-in "I genuinely don't know" escape hatch in the mechanics — every step, some token gets sampled from the distribution.

So when you ask about something the model never learned, or that doesn't exist, it doesn't hit a wall and stop. It does the only thing it can: produces the most plausible-looking continuation — which, for "cite a study on X," is a perfectly formatted citation… that may be entirely invented. The gap gets filled with confident-sounding fiction because confident-sounding text is exactly what it was trained to generate.

If the model is uncertain, why doesn't it just say so? A landmark 2025 analysis from OpenAI (Why Language Models Hallucinate) gives a sharp answer: standard training and evaluation reward guessing over admitting uncertainty.

Think of a student on a multiple-choice exam. A confident guess might be right (+1); writing "I don't know" is a guaranteed 0. Over millions of examples, that scoring teaches the optimal strategy: always answer confidently, never abstain. Most benchmarks grade models exactly this way — penalizing "I don't know" the same as a wrong answer — so models learn to bluff rather than express doubt.

The hopeful corollary: this is partly fixable by changing the incentives — reward calibrated uncertainty and allow abstention, and hallucination rates drop. Hallucination isn't pure mystery; it's partly a scoreboard we built wrong.

Hallucinations aren't uniform — they spike in predictable situations. Be on guard when:

• The question is past the knowledge cutoff — recent events the model never saw.
• The topic is rare or niche — sparse training data means the model is extrapolating.
• You ask for hyper-specific details — exact quotes, page numbers, statistics, citations, precise API signatures. These are easy to fabricate plausibly and hard for the model to actually recall.
• The prompt is ambiguous or under-specified — the model fills the gaps with assumptions.
• The context is very long — models pay less attention to the middle of a long context ("lost in the middle"), so they may ignore grounding you did provide.

Notice the pattern: hallucination is worst exactly where plausibility is easy but recall is hard.

Because it's inherent to the architecture, hallucination can't be fully eliminated — but it can be dramatically reduced. The main levers (each gets real depth later in the course):

• Grounding / RAG (the biggest lever). Retrieve relevant, verified documents and have the model answer from those instead of from memory. This is the single most effective fix — and it earns its own dedicated section later. (It's not magic: models can still ignore or misread retrieved text.)
• Permit and reward "I don't know." Prompt the model that abstaining is acceptable; ask it to cite or quote its sources so claims are checkable.
• Lower temperature for factual tasks — less creative wandering (a modest help, not a cure).
• Tools & verification — let the model look things up, run code, or check a database rather than recall from weights.
• Mind the cutoff — for anything recent, supply the facts; don't expect the model to know them.

The mindset shift: treat the model as a brilliant, fast, unreliable-on-facts writer — and build a system that supplies and checks the facts.

Hallucination is the #1 reason "the demo was magic but production embarrassed us." Internalize these:

• Never trust unverified factual output — especially citations, numbers, legal/medical claims, and API details. Confident tone is not evidence of correctness.
• Design for it from day one. For knowledge tasks, plan grounding/RAG; for high-stakes outputs, add verification or a human in the loop. Don't bolt safety on after a bad incident.
• Match the tool to the task. Need facts? Give the model the facts (retrieval/tools). Need prose/ideas? Its fluency is a feature.
• Set user expectations. UI should signal that outputs may be wrong (e.g. "verify important info"), and ideally show sources.
• It's a system problem, not just a model problem. The model won't stop hallucinating; your architecture is what makes the overall product trustworthy.

Provoke a hallucination. Ask a model: "Give me a peer-reviewed paper (with authors and year) about [a hyper-niche topic you invented]." Predict the failure mode.

→ A perfectly-formatted, completely fabricated citation — confident, plausible, fake. Why? The model optimizes plausibility, not truth, and there's no fact-check anywhere in the pipeline. This is why you never trust unverified citations/numbers — and why RAG exists.

• "Hallucination is a bug they'll soon patch." No — it's inherent to optimizing plausibility without a truth-check. It can be reduced, not eliminated.
• "The model is lying." No — no intent, no concept of truth. It emits the most plausible tokens, which sometimes happen to be false.
• "Confident = correct." No — fluency and confidence are produced by the same process for true and false statements alike.
• "Bigger/newer models don't hallucinate." They hallucinate less, but never zero — and they can sound more convincing when wrong.
• "Lowering temperature fixes it." Helps a little; the real fixes are grounding and verification, not a sampling knob.

• A hallucination is fluent, confident output that's false/unsupported — intrinsic (contradicts a given source) or extrinsic (fabricated).
• Root cause: models optimize plausibility, not truth, with no fact-check anywhere in the pipeline — so a confident lie and a fact look equally likely token-by-token.
• They must emit something, and training rewards confident guessing over "I don't know," so models bluff on gaps.
• Worst when plausibility is easy but recall is hard: post-cutoff, niche topics, specific citations/numbers, ambiguous or very long prompts.
• Reduce (not eliminate) with grounding/RAG, allowing abstention + citations, tools/verification, and the right model for the task.

That closes "How Models Generate Text." You now understand a model end-to-end: how it's built, aligned, and how it generates — including its failure modes. Next, the course pivots from theory to practice: Section 6 sets up your AI engineering stack and dev environment, starting with the anatomy of an LLM API call — and from here on, you build.