Source Summary: GPT-3
Full title: Language Models are Few-Shot Learners
Authors: Tom B. Brown et al. (45 authors total)
Affiliation: OpenAI
Venue: NeurIPS 2020 | arXiv: 2005.14165v4 (Jul 22, 2020)
PDF: papers/brown-2020-language-models-are-few-shot-learners.pdf
Contributions
- Scale as a path to few-shot learning — 175B parameter language model that performs tasks from a few in-context examples, without any gradient updates.
- In-context learning taxonomy — formal distinction between zero-shot, one-shot, and few-shot settings; demonstrated smooth scaling across all three.
- Broad task coverage — strong performance across translation, QA, cloze, arithmetic, word unscrambling, and novel reasoning tasks purely through prompting.
- Human-indistinguishable text generation — GPT-3-generated news articles rated ~52% human vs. ~88% human for actual articles.
Methodology
Model Architecture
- Same architecture as GPT-2: decoder-only Transformer, left-to-right autoregressive
- Modification: alternating dense and locally banded sparse attention (similar to Sparse Transformer)
- Context window: nctx = 2048 tokens
- 8 model sizes trained (125M → 175B), all for 300B tokens total
Model Sizes (Table 2.1)
| Model | Params | Layers | d_model | Heads |
|---|---|---|---|---|
| GPT-3 Small | 125M | 12 | 768 | 12 |
| GPT-3 Medium | 350M | 24 | 1024 | 16 |
| GPT-3 Large | 760M | 24 | 1536 | 16 |
| GPT-3 XL | 1.3B | 24 | 2048 | 24 |
| GPT-3 2.7B | 2.7B | 32 | 2560 | 32 |
| GPT-3 6.7B | 6.7B | 32 | 4096 | 32 |
| GPT-3 13B | 13B | 40 | 5140 | 40 |
| GPT-3 | 175B | 96 | 12288 | 96 |
Training Data
- Common Crawl (filtered, 410B tokens) — highest weight after quality filtering
- WebText2 (19B tokens)
- Books1, Books2 (~67B tokens)
- English Wikipedia (3B tokens)
- Total training: ~300B tokens (mix, not all unique)
- 3-step CC quality filtering: trained classifier, fuzzy deduplication, known-quality corpus filtering
Evaluation Settings
- Zero-shot: natural language instruction only, no examples
- One-shot: instruction + one demonstration
- Few-shot: instruction + K demonstrations (K ≈ 10–100, whatever fits in context window)
- No gradient updates in any setting — purely forward passes
Results
Language Modeling / Cloze
- Penn Treebank: 20.50 perplexity (SOTA at time)
- PTB zero-shot: comparable to fine-tuned BERT-LARGE
Question Answering
- TriviaQA: 71.2% (few-shot) — SOTA among closed-book models at time
- Natural Questions: 29.9% (few-shot)
- WebQA: 41.5% (few-shot)
SuperGLUE
- GPT-3 175B few-shot: 71.8 — close to fine-tuned BERT-LARGE (73.3)
Translation
- En→Fr: 32.6 BLEU (few-shot) — competitive with supervised systems
Arithmetic
- 3-digit addition: 80.2% (few-shot); 4-digit: 25.2%
- Shows emergent capability at large scale; near-zero for smaller models
Synthetic tasks (word manipulation)
- Cycle letters in word, anagram, reversed words: scale-dependent emergence
Key Design Choices
Why no fine-tuning?
- Fine-tuning exploits spurious correlations in narrow task distributions
- Performance may be inflated on benchmarks vs. true generalization
- Human intelligence is mostly zero/few-shot — the goal is to approach this
Scaling laws
- Loss decreases as smooth power law with model size (Kaplan et al., 2020)
- Few-shot performance gap over zero-shot grows with scale — larger models are better meta-learners
Data contamination
- Common Crawl may contain test sets → systematic contamination analysis
- Found minimal effect for most tasks; a few flagged with asterisk
Limitations
- Few-shot still lags fine-tuned on some tasks (ANLI, RACE, QuAC)
- Bidirectionality absent: autoregressive decoder can only attend left-to-right; weak on tasks requiring full-context understanding
- Interpretability: unclear whether model truly learns new tasks in-context or matches patterns from pre-training
- Compute: training estimated at hundreds of petaflop/s-days; not reproducible by most researchers
- Bias and fairness: gender, race, religion biases detected; model generates toxic completions under adversarial prompting
- No learning across episodes: in-context learning vanishes after the session
Key References
- Radford et al. 2019 — GPT-2 (architecture baseline)
- Kaplan et al. 2020 — Neural scaling laws
- Devlin et al. 2019 — BERT (fine-tuning paradigm being challenged)
- Child et al. 2019 — Sparse Transformer (attention pattern)