Reasoning Models: How o1 / R1 "Think"

Until now, every model we've discussed answers the way you'd blurt out a reflex: it reads your prompt and immediately starts producing the answer, one token at a time. Reasoning models — OpenAI's o-series, DeepSeek-R1, Claude's extended-thinking and Gemini's thinking modes — do something different. They think first: they generate a long internal train of thought, work the problem out step by step, sometimes catch and correct their own mistakes, and only then give you the answer.

This is the frontier of post-training, and it's the natural payoff of the last two lessons. Remember the online RL idea from DPO? Point that exact machinery at a different kind of reward — answers you can automatically check — and you get models that learn to reason. That's the whole trick, and it's beautifully simple once you see it.

You'll learn:

• What a reasoning model actually does differently (chain-of-thought)
• How they're trained: RL on verifiable rewards (RLVR) — and why no reward model is needed
• GRPO and DeepSeek-R1's famous "aha moment" — reasoning that emerged on its own
• The new scaling axis: test-time compute ("think longer") — and the overthinking trap
• When you should reach for a reasoning model (and when it's a costly mistake)

The difference is deliberation before answering. A standard model is like answering instantly from your gut; a reasoning model is like working it out on scratch paper first.

Concretely, before the visible answer the model produces a long chain-of-thought (CoT) — a stream of intermediate "thinking" tokens where it plans, tries approaches, checks itself, and backtracks. Then it writes the final answer.

Q: A bat and a ball cost $1.10. The bat costs $1.00 more
   than the ball. How much is the ball?

Standard model:   "$0.10."        ← fast, and wrong

Reasoning model:  <thinking> If the ball is x, the bat is
   x+1.00, so 2x+1.00 = 1.10 → 2x = 0.10 → x = 0.05.
   Check: 0.05 + 1.05 = 1.10. ✓ </thinking>
   "The ball costs $0.05."        ← deliberated, and right

That hidden deliberation is why reasoning models crush hard math, multi-step logic, and complex coding that trip up standard models. (Providers usually hide or summarize the raw chain-of-thought, but it's there, and you pay for those tokens.)

Here's the key idea, and it builds directly on RLHF/DPO. Those methods needed a reward model to guess what humans prefer — because "is this a good essay?" is subjective. But for a huge class of problems, you don't need to guess — you can just check:

• Math problem → is the final answer correct?
• Coding task → do the unit tests pass?
• Logic puzzle → does the solution satisfy the constraints?

This is RLVR — Reinforcement Learning with Verifiable Rewards. Swap the subjective, learned reward model for a deterministic verifier that automatically scores answers as right or wrong. Then run the same online-RL loop: the model attempts the problem, the verifier checks it, and the model is rewarded for getting it right. No human labels, no preference model, no reward hacking the correctness signal — the answer is simply right or it isn't.

The RL algorithm that made this famous is GRPO (Group Relative Policy Optimization), from DeepSeek. Its clever simplification over PPO: instead of training a separate value model to judge "how good is this step," GRPO has the model generate a whole group of answers to the same problem, scores them all with the verifier, and uses the group's average as the baseline. Answers that beat the group average get reinforced; below-average ones get suppressed. Simpler, cheaper, no critic model.

The stunning result was DeepSeek-R1-Zero: they ran this RL loop on a base model with no SFT — and sophisticated reasoning emerged on its own. With nothing but "get the answer right" as a reward, the model spontaneously learned to write long chains of thought, try multiple approaches, and re-examine and correct its own work — the now-famous "aha moment."

The profound lesson: you don't have to teach a model how to reason by example. If you reward correct answers and give it room to think, it discovers reasoning by itself.

Remember scaling laws — bigger models + more data + more training compute = better? Reasoning models opened a second, independent axis: spend more compute at inference time.

The insight: for a hard problem, letting the model think longer — generate a longer chain-of-thought before answering — reliably improves the answer, without changing the model's weights at all. This is test-time (inference-time) scaling, and it's genuinely new physics for the field: you can trade latency and dollars at runtime for accuracy.

You control it with a thinking budget — a cap on how many reasoning tokens the model may use (Google's Gemini exposes a literal thinkingBudget; Anthropic uses a thinking block with budget_tokens; OpenAI a reasoning-effort setting). Bigger budget → more deliberation → better on hard tasks, at higher cost and latency. It's a dial you tune per task — one of the most practical levers in modern AI engineering.

Here's the nuance most hype skips: thinking longer is not monotonically better. Research on test-time scaling finds a curve that rises and then falls — past some point, extra thinking hurts:

• Overthinking: on easy questions, a reasoning model can spiral into long, tangential deliberation, burning tokens and sometimes talking itself out of the right answer.
• Wasted cost & latency: a question a standard model answers instantly might cost 10× the tokens and seconds of delay on a reasoning model — for no benefit.
• The budget bind: capping thinking too hard causes underthinking on genuinely difficult problems; too loose causes overthinking on easy ones. Right-sizing the budget is a real skill.

An increasingly popular alternative is parallel thinking / self-consistency: instead of one very long chain, sample several independent shorter solutions and take the majority answer — often more accurate per unit of compute than simply thinking longer.

RL-training a reasoning model is expensive — but once you have one, there's a shortcut to spread the capability: distillation (which you met in the SFT lesson).

Take a strong reasoning model, have it produce thousands of high-quality chains-of-thought, and then SFT a much smaller model on those reasoning traces. The small model learns to imitate the big one's thinking. DeepSeek released exactly this — small distilled R1 variants that reason far above their weight class.

The payoff: you don't need a giant frontier model (or a giant RL run) to get useful step-by-step reasoning. A well-distilled small model can deliver a lot of it cheaply — which matters enormously when you're paying per token in production.

Reasoning models are a tool with a specific job — using them well is pure AI-engineering judgment:

• Use them for genuinely hard, multi-step work: tough math, complex/agentic coding, planning, intricate logic. (Recall the three flavors lesson — reasoning is the ~5% case, not your default.)
• Don't use them for simple tasks — extraction, classification, basic chat. You'll pay far more, wait far longer, and risk overthinking for zero gain.
• Budget the thinking. Treat the thinking budget / effort setting as a real knob: more for hard problems, less (or off) for easy ones.
• Mind cost & latency: you're billed for hidden reasoning tokens you never see — reasoning answers can cost and take 5–20× a standard reply.
• Hybrid is the trend: increasingly it's one model with a thinking toggle (think hard when needed, answer fast otherwise) rather than separate products — so "when do I turn thinking on?" becomes a per-request decision you'll make constantly.

When is 'thinking' worth it? You have a reasoning model. For each task, predict: does extended thinking help, or just waste tokens/latency?

1. "What's 2+2?" → ? 2. "Prove this is a valid red-black tree after these 5 insertions." → ? 3. "Extract the email from this text." → ?

→ 1 & 3: waste (trivial — thinking adds cost and risks overthinking). 2: help (genuinely multi-step). The skill is reserving reasoning for the hard ~5%.

• "Reasoning models are just bigger." No — the gain comes from training (RL on verifiable rewards) + thinking at inference, not raw size. A distilled small model can reason well.
• "They were taught to reason step-by-step with examples." Largely no — RLVR rewards correct answers and reasoning emerges (R1-Zero did it with no SFT at all).
• "More thinking always helps." No — overthinking is real; accuracy rises then falls, and easy tasks get worse and pricier.
• "Reasoning needs a fancy reward model." The opposite — its power is a simple verifier (right/wrong), no preference model needed.
• "Always use the reasoning model to be safe." No — it's the wrong default: slow and expensive. Match the model to the task's difficulty.

Before the final recap, step back and see the whole journey at once. Every lesson in this section was a chapter in one story — how the field turned a raw, knowledgeable-but-useless base model into a helpful, safe, reasoning assistant. Here is that story told through the actual models that pioneered each method:

Read left to right, it's the exact arc of this section: GPT-3 could only predict text; InstructGPT first aligned it with SFT + RLHF; ChatGPT brought RLHF-tuned assistants to the world; Claude aligned from a written constitution (Constitutional AI / RLAIF); the DPO era made preference tuning simple enough for anyone; and o1 / R1 added reasoning via RL on verifiable rewards. Same through-line, six milestones — and you now understand the method behind each one.

• Reasoning models think before they answer — generating a long internal chain-of-thought, then the final response — which is why they excel at hard math, code, and logic.
• They're trained with RLVR (RL on Verifiable Rewards): replace the subjective reward model with a deterministic verifier (answer correct? tests pass?) and run online RL.
• GRPO made this cheap (group-relative, no critic), and DeepSeek-R1-Zero showed reasoning can emerge from pure RL — the "aha moment."
• A new scaling axis appeared: test-time compute — "think longer" via a thinking budget — but beware overthinking (more isn't always better).
• Engineering judgment: use reasoning models for genuinely hard multi-step tasks, budget the thinking, and mind the real cost/latency of hidden reasoning tokens.

That closes Post-Training & Alignment — you now understand the full journey from raw base model to aligned, preference-tuned, reasoning assistant. Next section: we open the hood on how a model actually generates text — turning raw scores into the next token — starting with logits → probabilities via softmax.