Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Authors: Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.
Year: 2022 | NeurIPS
Source: wei-2022-chain-of-thought-prompting-elicits-reasoning-in
What it is
This paper showed that providing a few examples of step-by-step reasoning in the prompt dramatically improves LLM performance on multi-step reasoning tasks — without any fine-tuning. More importantly, it established that this capability is emergent: it only appears in models above ~100B parameters, not in smaller models. Chain-of-thought prompting became one of the most influential techniques in practical LLM deployment and reopened the debate about whether neural networks engage in genuine symbolic reasoning.
The technique
Standard few-shot prompting gives the model (question, answer) pairs:
Q: Roger has 5 tennis balls. He buys 2 cans of 3. How many does he have?
A: 11.
Chain-of-thought prompting gives (question, reasoning, answer) triples:
Q: Roger has 5 tennis balls. He buys 2 cans of 3. How many does he have?
A: Roger started with 5. 2 cans × 3 = 6 additional. 5 + 6 = 11. The answer is 11.
When 8 such examples are included in the prompt, the model generates its own reasoning chains for new questions before producing answers.
Results
| Benchmark | Type | PaLM 540B standard | PaLM 540B + CoT |
|---|---|---|---|
| GSM8K | Math word problems | 18% | 57% |
| SVAMP | Math variations | 46% | 63% |
| AQuA | Algebraic | 33% | 47% |
| StrategyQA | Commonsense | 65% | 73% |
| Date Understanding | Symbolic | 49% | 62% |
PaLM 540B with CoT achieved 57% on GSM8K — surpassing fine-tuned GPT-3 with a verifier (55%). The gains are striking and consistent across reasoning types.
Emergence with scale
The most important finding is not the absolute numbers but the scaling profile:
Model size: 6.7B 13B 175B 540B
Standard: 5% 7% 17% 18%
CoT: 3% 5% 20% 57% ← CoT only helps at large scale
Small models produce worse performance with CoT — they generate incoherent reasoning chains that lead to wrong answers. The reasoning capability appears discontinuously at ~100B parameters. This is the definition of an emergent capability: not a smooth scaling curve, but a threshold phenomenon.
The symbolic vs. connectionist interpretation
CoT raises a direct question about the nature of LLM reasoning.
Symbolic interpretation: the model externally represents intermediate symbolic states — equations, named quantities, logical steps — and reasons over them. Each step is derived from the previous, mimicking the Advice Taker’s inference chains (McCarthy 1958). The reasoning is genuinely compositional.
Connectionist interpretation: CoT is a prompt format that shifts the model’s output distribution toward token sequences that resemble systematic reasoning. The model is pattern-matching on the format of reasoning, not “actually thinking.” The chains look correct because training data contained many correct reasoning chains.
Evidence favors a nuanced middle ground:
- CoT chains are inspectable and often catch/avoid errors
- CoT fails on very long chains and unfamiliar reasoning patterns
- CoT produces wrong-but-plausible-looking chains (“hallucinated reasoning”)
The practical result: for tasks within the model’s training distribution, CoT reasoning is highly reliable. For novel tasks requiring compositional generalization, it degrades.
Zero-shot CoT
Kojima et al. (2022), building directly on this paper, showed that simply appending “Let’s think step by step.” to a question (no examples needed) triggers chain-of-thought reasoning in large models:
Q: [hard question]
A: Let's think step by step.
[model generates steps]
Therefore, the answer is [X].
This works for models ≥~100B parameters. It suggests the reasoning pattern is latent in the model’s weights — CoT examples are not teaching the model a new skill, they’re activating one it already has.
Connection to Newell & Simon’s cognitive architecture
The external context window acting as a scratchpad for intermediate steps is structurally identical to the working memory in Newell and Simon’s problem-solving architecture (newell-1976-computer-science-as-empirical-inquiry): the model maintains a state representation, applies operators (reasoning steps) to it, and tracks progress toward a goal.
Whether this means LLMs are implementing Physical Symbol Systems implicitly — or just mimicking them statistically — remains the central unresolved question of LLM theory.
See also
- brown-2020-language-models-are-few-shot-learners — GPT-3: the few-shot capability CoT builds on
- kaplan-2020-scaling-laws-for-neural-language — Scaling laws: context for why CoT is emergent
- mccarthy-1958-programs-with-common-sense — Advice Taker: the symbolic system CoT approximates
- newell-1976-computer-science-as-empirical-inquiry — Physical Symbol Systems: the theoretical framework for interpreting CoT