Chain-of-Thought: Giving Models Time to Think

Welcome to advanced prompting. We open with the technique that started it all: Chain-of-Thought (CoT) — simply asking the model to reason step by step before giving its answer.

Why does this help so much? Because a model generates one token at a time, left to right. Ask a hard, multi-step question and demand an immediate answer, and the model is forced to commit to its first word before it has 'worked anything out' — like a student blurting the first number that comes to mind. CoT gives it room to do the work on the page first. The accuracy jump on reasoning tasks can be dramatic.

You'll also see how this connects to the reasoning models from earlier: CoT prompting is the manual version of what o1 and R1 learned to do automatically — and knowing the difference saves you from a common 2026 mistake.

You'll learn:

• Why models fail by answering before thinking
• Zero-shot CoT ('think step by step') and few-shot CoT
• The crucial difference between CoT prompting and reasoning models
• When CoT helps (and when it just wastes latency)
• The catch: cost and unfaithful reasoning

Consider a simple multi-step question:

"A shirt costs $40 after a 20% discount. What was the original price?"

Asked for a direct answer, models often blurt **"$48"** — naively adding 20$40. Wrong. The correct path is original × 0.8 = 40 → original = $50. The model knows how to do this — but forced to answer immediately, it grabbed the most superficially plausible number instead of reasoning.

This is the core issue: the first tokens of a direct answer are generated before any reasoning has happened. For lookups that's fine; for anything multi-step, it's like being made to shout an answer before you've thought. CoT removes that constraint.

CoT prompting asks the model to lay out its reasoning before the final answer. Those intermediate steps aren't decoration — they're the mechanism:

• The model uses its sequential generation to actually work the problem out, one step at a time, instead of leaping to a conclusion.
• Each step it writes becomes context the next step can attend to — so by the time it states the answer, the reasoning is already 'on the page' for it to read from.

In effect, CoT lets the model spend more computation (more tokens) on a hard problem — thinking out loud rather than in a single leap. Watch the same question, direct vs. with CoT:

The simplest, highest-ROI version needs no examples at all — just append a trigger phrase:

Q: A shirt costs $40 after a 20% discount. What was the original price?
A: Let's think step by step.

That one line — "Let's think step by step" — was a landmark finding (Kojima et al., 2022): it reliably unlocks step-by-step reasoning on arithmetic, logic, and commonsense problems, without any worked examples. Variations like 'Show your reasoning before answering' or 'Work through this carefully' do the same job.

This is the first thing to reach for when a model fumbles a multi-step task: before rewriting your whole prompt, just ask it to think first. It's nearly free to try and often fixes the problem outright.

When zero-shot CoT isn't enough — or you want the reasoning to follow a specific style or method — use few-shot CoT: provide examples whose answers include the worked reasoning, not just the final result (combining few-shot from the last section with CoT).

Q: ...   A: First I ... then I ... so the answer is X.
Q: ...   A: First I ... then I ... so the answer is Y.
Q: {your question}   A:

This demonstrates the reasoning pattern you want the model to imitate (great for domain-specific logic, e.g. 'always check units, then compute'). One honest 2026 caveat: modern models have absorbed so much CoT data in training that few-shot reasoning exemplars are increasingly redundant — zero-shot CoT often matches them. So try zero-shot first, and reach for few-shot CoT only when you need to enforce a particular reasoning style.

This is the distinction that trips people up in 2026. There are two routes to step-by-step reasoning:

• CoT prompting — you ask a standard model to reason step by step. The reasoning is visible in the output, and you control it via the prompt.
• Reasoning models (o1, R1, Claude extended-thinking) — the model was trained to reason internally before answering (the RLVR lesson). It does CoT automatically, usually in a hidden 'thinking' phase.

The practical rule: don't add 'think step by step' to a reasoning model. It already does — so the instruction is redundant, and worse, it can interfere with the model's own tuned process and degrade results (providers explicitly advise against it). Use CoT prompting for standard models; for reasoning models, just ask the question (and control depth via the thinking budget instead). Same destination — built in vs prompted in — but you don't do both.

CoT is a targeted tool, not a universal upgrade. Research is clear that the gains concentrate on reasoning-heavy tasks:

• ✅ Big help: multi-step math, symbolic/logic problems, commonsense reasoning, multi-constraint planning, complex analysis.
• ❌ Little or no help: simple lookup/retrieval, classification, formatting, single-step factual answers. Here CoT just adds tokens, latency, and cost for no accuracy gain — and can even add noise.

So don't slap 'think step by step' on every prompt. Use it where the task genuinely requires chaining multiple steps; skip it for straightforward ones. (This mirrors the three flavors lesson: reserve heavy reasoning for genuinely hard work.)

Two limitations every practitioner should know:

• It costs latency and money. CoT means generating many more tokens before the answer — and output tokens are the expensive, slow part (Iron Triangle + pricing lessons). For high-volume or real-time paths, reserve CoT for the hard queries, or use a compact 'brief reasoning' instruction.
• The reasoning can be unfaithful. Unsettling but important: the chain of thought a model writes is not guaranteed to be the real reason it reached its answer. Models can produce plausible-sounding reasoning that post-hoc rationalizes a conclusion they'd have reached anyway — research shows they 'sometimes lie about their reasoning.' So don't treat the visible CoT as a trustworthy audit trail or a guarantee of correctness; it improves answers, but it isn't a window into the model's 'true' process.

And because a single chain can still go wrong, the natural next step is to generate several and compare them — which is exactly the next lesson, self-consistency.

• Default move: add 'Let's think step by step' (or 'show your reasoning first') when a standard model fumbles a multi-step task.
• Keep the answer parseable. If you need a clean result, separate reasoning from answer with structure: reasoning in <thinking>…</thinking>, the result in <answer>…</answer> — then extract just the answer (delimiters lesson).
• Reserve it for hard tasks to avoid paying latency/cost where it doesn't help.
• On reasoning models, don't add CoT — ask directly and tune the thinking budget.
• For the hardest problems, combine CoT with self-consistency (sample multiple chains, take the majority) — next lesson.

Feel the difference. Take a multi-step word problem and a standard model. Ask it directly → note if it's wrong. Now add 'Let's think step by step.' → watch it reason to the right answer.

Then the gotcha: try the same phrase on a reasoning model (o1/R1/extended-thinking). Why is it pointless (or even harmful)? → It already reasons internally; the instruction is redundant and can interfere. CoT prompting is for standard models.

• "CoT makes the model smarter." It lets the model use its ability by reasoning before answering — it doesn't add knowledge; it prevents premature commitment.
• "Add 'think step by step' to every prompt." No — it helps on multi-step reasoning, and just wastes tokens/latency on simple tasks.
• "Use CoT prompting on reasoning models too." No — they reason internally; explicit CoT is redundant and can hurt them.
• "The chain of thought is the model's real reasoning." Not reliably — it can be a post-hoc rationalization; don't trust it as an audit.
• "Few-shot CoT is always better than zero-shot." Not anymore — modern models often do as well zero-shot; try that first.

• CoT prompting asks a model to reason step by step before answering, letting it work the problem out instead of committing to a first-token guess — big gains on multi-step tasks.
• Zero-shot CoT ('Let's think step by step') is the simplest, highest-ROI move; few-shot CoT shows worked reasoning when you need a specific style (often redundant on modern models — try zero-shot first).
• CoT prompting ≠ reasoning models: standard models you prompt to reason; reasoning models reason internally by training — don't add 'think step by step' to them.
• Use it for math/logic/multi-step tasks; skip it for simple lookups (it costs latency & tokens for nothing).
• Caveats: it adds cost/latency, and the visible chain can be unfaithful — not a reliable audit of the model's real reasoning.

Next: one chain can still be wrong — so self-consistency generates several reasoning paths and takes the majority answer for a reliability boost on hard problems.