GPT-1 — Improving Language Understanding by Generative Pre-Training
Radford et al., OpenAI, 2018
Generative pre-training on a large unlabeled corpus followed by discriminative fine-tuning on each specific task.
Core Idea
GPT-1 established the pre-train → fine-tune paradigm for NLP: train a large Transformer decoder autoregressively on massive text, then adapt to downstream tasks with minimal changes. The insight is that language modeling forces the model to learn rich world knowledge and linguistic structure, which transfers broadly.
Architecture
- 12-layer Transformer decoder (causal / left-to-right attention only)
- 117M parameters, 768 hidden dim, 12 heads, 3072 FFN dim
- Pre-trained on BooksCorpus (~800M words)
- Context window: 512 tokens
Two-Stage Training
Stage 1: Unsupervised Pre-training
Standard causal language modeling on token sequences :
Stage 2: Supervised Fine-tuning
Given labeled pairs , predict label from final hidden state:
Adding the LM objective as auxiliary loss improves generalization and convergence.
Input Format for Tasks
All tasks represented as linear token sequences — no new heads needed:
| Task | Format |
|---|---|
| Classification | [s] text [e] |
| Entailment | [s] premise $ hypothesis [e] |
| Similarity | [s] text₁ $ text₂ [e] (both orders) |
| QA | [s] context $ question $ answer_i [e] |
Results
State-of-the-art on 9/12 tasks:
- +8.9% commonsense reasoning (Stories Cloze)
- +5.7% QA (RACE)
- +1.5% NLI (MultiNLI)
Relationship to Other Work
| Paper | Relationship |
|---|---|
| Vaswani 2017 (Transformer) | Architecture basis — GPT uses Transformer decoder blocks |
| Devlin 2019 (BERT) | Successor — adds bidirectional masking, outperforms GPT-1 on most tasks |
| Brown 2020 (GPT-3) | Direct descendant — scales to 175B, achieves few-shot without fine-tuning |
| Hu 2021 (LoRA) | More efficient fine-tuning for GPT-style models |
Key Insight
The unidirectional language model objective, despite being simpler than BERT’s masked LM, scales remarkably well (GPT-2, GPT-3, GPT-4 follow the same paradigm). The choice of decoder-only architecture proved crucial for generation tasks.