GPT-3: Language Models are Few-Shot Learners

Authors: Tom B. Brown et al. (OpenAI, 45 authors)
Year: 2020 | NeurIPS
Source: brown-2020-language-models-are-few-shot-learners

What it is

GPT-3 is a 175-billion-parameter autoregressive language model that can perform new tasks from a handful of in-context examples — with no weight updates, no fine-tuning, and no task-specific training data. It is the paper that demonstrated scaling as a path to general-purpose language understanding.

The core idea: in-context learning

Prior work (BERT, fine-tuned GPT-2) required thousands of labeled examples to adapt a model to a new task. GPT-3’s bet: make the model large enough that it can learn from the prompt itself.

Zero-shot:  "Translate English to French: cheese =>"
One-shot:   "sea otter => loutre de mer\ncheese =>"
Few-shot:   [K examples in context] "cheese =>"

All three settings use only forward passes — no gradient updates ever occur at inference.

Scale

GPT-3 is 10× larger than any previous non-sparse language model at time of publication:

Model	Parameters
BERT-LARGE	340M
GPT-2 XL	1.5B
Megatron-LM	8.3B
T5-11B	11B
GPT-3	175B

Architecture: decoder-only Transformer (same as GPT-2) with 96 layers, d_model=12288, 96 attention heads, context window 2048 tokens. Trained on ~300B tokens across Common Crawl, WebText2, Books, and Wikipedia.

Zero-shot, one-shot, few-shot

The paper formalizes three evaluation settings that have become standard:

Zero-shot: instruction only — closest to how humans work from descriptions
One-shot: one example — matches how Mechanical Turk tasks are communicated to workers
Few-shot: K examples (typically 10–100) filling the context window — best performance, but needs examples

Key finding: the gap between zero-shot and few-shot grows with model size — larger models are better meta-learners.

Results

Task	GPT-3 (few-shot)	Prior SOTA
TriviaQA (closed-book)	71.2%	68.0% (fine-tuned)
WebQA	41.5%	37.4%
SuperGLUE	71.8	89.3 (T5 fine-tuned)
En→Fr translation	32.6 BLEU	competitive
3-digit arithmetic	80.2%	—

GPT-3 matches or exceeds fine-tuned SOTA on some tasks; substantially behind on others (ANLI, RACE, reading comprehension requiring long context).

Human-distinguishability test: GPT-3-generated news articles rated as human ~52% of the time, vs. ~88% for actual articles.

Scaling laws connection

Kaplan et al. (2020) showed that loss follows a smooth power law as a function of compute, data, and model size. GPT-3 validates this: the 175B model achieves better few-shot performance than all smaller variants, and the improvement is roughly predictable from the scaling curves.

Why not fine-tune?

Fine-tuning risks:

Overfitting to narrow task distributions — models memorize spurious features
Benchmark inflation — fine-tuned performance may overestimate true generalization
Label cost — requires thousands of labeled examples per task

GPT-3 sacrifices peak per-task accuracy for task-agnostic generality, matching how humans apply background knowledge to new problems.

Limitations

Still lags fine-tuned models on structured tasks (ANLI: 40.2% vs. 72% for RoBERTa)
Left-to-right only: no bidirectional context; weak on fill-in tasks where BERT excels
Data contamination: Common Crawl may contain test benchmarks; authors flag affected results
Cost: training estimated at ~$4.6M; not reproducible
Bias: gender and religious stereotypes present in completions; requires mitigation for deployment
No persistent learning: in-context examples are forgotten after the session

Historical significance

GPT-3 shifted the field’s intuition: pre-training + prompting became a viable alternative to pre-training + fine-tuning. This directly seeded:

InstructGPT / ChatGPT (RLHF on GPT-3)
Prompt engineering as a discipline
The “foundation model” framing (Bommasani et al., 2021)
GPT-4, Claude, Gemini — all scaling the same decoder paradigm

AI/ML Wiki

Explorer

brown-2020-language-models-are-few-shot-learners