Source Summary: Wei et al. 2022 — Chain-of-Thought Prompting
Full title: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Authors: Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou
Affiliation: Google Brain
Venue: NeurIPS 2022 (arXiv: 2201.11903)
PDF: papers/wei-2022-chain-of-thought-prompting-elicits-reasoning-in.pdf
Contributions
- Chain-of-thought prompting — providing a few examples of step-by-step reasoning in the prompt unlocks complex multi-step reasoning in large language models.
- Emergent reasoning capability — CoT reasoning only appears in models above ~100B parameters; it is an emergent property of scale, not present in smaller models.
- Broad applicability — demonstrated on arithmetic, commonsense, and symbolic reasoning benchmarks, covering very different types of multi-step reasoning.
- Zero-shot CoT — follow-up work showed “Let’s think step by step” alone (without examples) triggers reasoning in large models.
The Key Idea
Standard few-shot prompting provides (input, output) pairs:
Q: Roger has 5 tennis balls. He buys 2 cans of 3. How many does he have?
A: 11.
Q: [new question]
A: ?
Chain-of-thought prompting provides (input, reasoning, output) triples:
Q: Roger has 5 tennis balls. He buys 2 cans of 3. How many does he have?
A: Roger started with 5 balls. 2 cans × 3 balls = 6 balls. 5 + 6 = 11.
The answer is 11.
Q: [new question]
A: [model generates step-by-step reasoning, then answer]
The model learns to externalize intermediate reasoning steps before committing to a final answer.
Key Results
| Benchmark | Task type | Standard prompting | CoT prompting |
|---|---|---|---|
| GSM8K | Math word problems | 18% (PaLM 540B) | 57% (PaLM 540B) |
| SVAMP | Math word problems | 46% | 63% |
| AQuA | Algebraic | 33% | 47% |
| StrategyQA | Commonsense | 65% | 73% |
| Date Understanding | Symbolic | 49% | 62% |
PaLM 540B with CoT: 57% on GSM8K — surpassing even fine-tuned GPT-3 with a verifier (55%).
Emergence with Scale
Critical finding: CoT helps only in large models.
GPT-3 6.7B: CoT → same or worse performance
GPT-3 175B: CoT → modest improvement
PaLM 540B: CoT → large improvement, state-of-the-art
Smaller models don’t benefit from chain-of-thought — they produce incoherent or incorrect reasoning chains. This makes CoT an emergent capability: it appears discontinuously at sufficient scale.
Why CoT Works: Interpretations
Symbolic interpretation: CoT forces the model to explicitly construct intermediate symbolic representations (equations, logical steps, named quantities) before producing an output. Each step grounds the next.
Connectionist interpretation: CoT is a clever prompt format that shifts the model’s output distribution toward tokens that resemble systematic reasoning — the model is pattern-matching on the format, not “actually thinking.”
The paper doesn’t resolve this debate, but notes that CoT chains are inspectable and often catch errors in intermediate steps — suggesting they capture real reasoning structure, not just surface patterns.
Connection to Symbolic AI
The chain-of-thought pattern is structurally identical to the Advice Taker’s inference chains (McCarthy 1958): a sequence of logical derivations, each following from the previous, leading to an action.
The key difference: in symbolic AI, the inference engine is explicit and guaranteed correct; in CoT, the inference is emergent in the neural network and probabilistically correct. CoT demonstrates that LLMs can approximate symbolic reasoning without explicit symbolic machinery — but the approximation fails for small models and for very long chains.
Key References (from paper)
- Brown et al. 2020 (GPT-3) — few-shot prompting baseline
- Kaplan et al. 2020 — scaling laws (context for emergence)
- Cobbe et al. 2021 — fine-tuned GPT-3 with verifier on GSM8K (outperformed by CoT)
- Kojima et al. 2022 — “Let’s think step by step” (zero-shot CoT follow-up)