Source — Improving Language Understanding by Generative Pre-Training

Source — Improving Language Understanding by Generative Pre-Training (GPT-1)

Authors: Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever
Year: 2018
Institution: OpenAI

---

Contributions

1. Two-stage training paradigm: Unsupervised generative pre-training on large corpus followed by supervised fine-tuning on task-specific data.
2. Task-agnostic model: Single pre-trained Transformer decoder that can be fine-tuned with minimal architectural changes across diverse NLP tasks.
3. Demonstrated transfer learning for NLP: Showed that large-scale language model pre-training substantially improves NLP benchmarks (NLI, QA, commonsense reasoning, semantic similarity).

---

Methodology

Architecture

• 12-layer Transformer decoder (117M parameters)
• Causal (left-to-right) language modeling objective
• 768-dimensional hidden states, 12 attention heads
• Trained on BooksCorpus (~7,000 unpublished books, ~800M words)

Pre-training Objective

Maximize likelihood of tokens given preceding context:

$L_1(\mathcal{U})={\sum }_i\log P(u_i|u_{i-k},\dots ,u_{i-1};\Theta )$

Fine-tuning

For labeled dataset with input tokens $x^1,\dots ,x^m$ and label $y$:

$L_2(\mathcal{C})={\sum }_{(x,y)}\log P(y|x^1,\dots ,x^m)$

Combined objective: $L_3(\mathcal{C})=L_2(\mathcal{C})+\lambda \cdot L_1(\mathcal{C})$

Task-Specific Input Transformations

Tasks structured as sequences using delimiter tokens, avoiding major architectural changes:

• Classification: [Start] Text [Extract]
• Entailment: [Start] Premise [Delim] Hypothesis [Extract]
• Similarity: both orderings concatenated
• QA/Commonsense: [Start] Context [Delim] Answer_i [Extract]

---

Results

• Achieved state-of-the-art on 9 of 12 NLP tasks evaluated
• +8.9% on commonsense reasoning (Stories Cloze)
• +5.7% on question answering (RACE)
• +1.5% on textual entailment (MultiNLI)
• Transfer from language model outperformed task-specific supervised models

---

Limitations

• Unidirectional: only left-to-right context (addressed by BERT with bidirectional masking)
• Relatively small scale by later standards (117M parameters)
• BooksCorpus domain may limit generalization
• Fine-tuning still required per task

---

Significance

GPT-1 established the pre-train/fine-tune paradigm that became the dominant approach in NLP, directly inspiring BERT, GPT-2, GPT-3, and the entire LLM ecosystem.