Source Summary: BERT
Full title: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Authors: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova
Affiliation: Google AI Language
Venue: NAACL 2019 | arXiv: 1810.04805v2 (May 24, 2019)
PDF: papers/devlin-2019-bert-pre-training-of-deep-bidirectional.pdf
Contributions
- Bidirectional pre-training via Masked Language Model (MLM) — first deep bidirectional Transformer for NLP pre-training.
- Next Sentence Prediction (NSP) — auxiliary pre-training task for sentence-pair understanding.
- Universal fine-tuning — single pre-trained model fine-tuned with minimal modifications for 11 NLP tasks, all achieving SOTA.
- GLUE score: 80.5% (+7.7pp); SQuAD v1.1 F1: 93.2 (+1.5pp); SQuAD v2.0 F1: 83.1 (+5.1pp).
Methodology
Model Architecture
- Multi-layer bidirectional Transformer encoder (Vaswani et al. 2017)
- BERT-BASE: L=12, H=768, A=12 → 110M params
- BERT-LARGE: L=24, H=1024, A=16 → 340M params
- BERT-BASE chosen to match OpenAI GPT size for fair comparison
Input Representation
Input = Token Embeddings + Segment Embeddings + Position Embeddings
- WordPiece tokenization, 30K vocabulary
- Special tokens:
[CLS](start, used for classification),[SEP](sentence separator) - Segment embeddings: A/B to distinguish sentence pairs
Pre-training Task 1: Masked LM (MLM)
- Randomly mask 15% of tokens; predict original token from context
- Why not always
[MASK]: pre-train/fine-tune mismatch → 80% replace with[MASK], 10% random token, 10% unchanged - Enables true bidirectional conditioning (unlike GPT’s left-to-right)
Pre-training Task 2: Next Sentence Prediction (NSP)
- Binary classification: is sentence B the actual next sentence after A?
- 50% IsNext, 50% NotNext (random sentence)
- Trains inter-sentence understanding for QA and NLI tasks
Pre-training Data
- BooksCorpus (800M words) + English Wikipedia (2,500M words)
- Document-level corpus (not shuffled sentences) for long contiguous sequences
Fine-tuning
- Plug task-specific input/output into BERT, fine-tune all parameters end-to-end
- Classification: use
[CLS]vector → linear layer - Token-level (NER, QA): use per-token vectors → output layer
- Fine-tuning cost: ~1 hour on Cloud TPU from same pre-trained checkpoint
Results
GLUE (Table 1)
| System | MNLI | QQP | QNLI | SST-2 | Average |
|---|---|---|---|---|---|
| OpenAI GPT | 82.1 | 70.3 | 87.4 | 91.3 | 75.1 |
| BERT-BASE | 84.6 | 71.2 | 90.5 | 93.5 | 79.6 |
| BERT-LARGE | 86.7 | 72.1 | 92.7 | 94.9 | 82.1 |
Official GLUE score: 80.5 (BERT-LARGE)
SQuAD
| System | v1.1 EM | v1.1 F1 | v2.0 F1 |
|---|---|---|---|
| Best ensemble (Dec 2018) | 86.0 | 91.7 | — |
| BERT-LARGE single | 84.1 | 90.9 | 83.1 |
| BERT-LARGE ensemble+TriviaQA | 87.4 | 93.2 | — |
Key Design Choice: Bidirectionality
GPT (left-to-right): each token attends only to previous tokens
BERT (bidirectional): each token attends to all tokens in all layers
For tasks like QA, seeing both left and right context is critical. A question about “the capital of France” needs to see “Paris” whether it appears before or after the question word.
Limitations
[MASK]token never seen at fine-tuning → pre-train/fine-tune mismatch (partially mitigated)- NSP later shown to be of limited benefit (RoBERTa, 2019: removing NSP improves or matches)
- Encoder-only: cannot directly generate text (unlike GPT)
- Large compute for pre-training (4 days on 64 TPUs for BERT-LARGE)
Key References
- Vaswani et al. 2017 — Transformer architecture
- Radford et al. 2018 — OpenAI GPT (left-to-right pre-training baseline)
- Peters et al. 2018 — ELMo (feature-based bidirectional representation)