BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Authors: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova (Google AI Language)
Year: 2019 | NAACL
Source: devlin-2019-bert-pre-training-of-deep-bidirectional
What it is
BERT is a pre-trained Transformer encoder that reads text bidirectionally — each token attends to all other tokens simultaneously. Trained on masked language modeling and next-sentence prediction, a single BERT checkpoint can be fine-tuned for 11 NLP tasks with minimal architectural changes.
The core idea: bidirectional pre-training
Prior language models processed text left-to-right (GPT) or concatenated separate left-to-right and right-to-left models (ELMo). BERT’s key insight: mask some input tokens and predict them from all surrounding context — forcing the model to learn true bidirectional representations in a single encoder.
Input: The [MASK] sat on the mat
Output: model predicts "cat" using context from both directions
This contrasts with GPT, where “cat” can only attend to “The” when being predicted.
Architecture
| Variant | Layers | Hidden | Heads | Parameters |
|---|---|---|---|---|
| BERT-BASE | 12 | 768 | 12 | 110M |
| BERT-LARGE | 24 | 1024 | 16 | 340M |
Input representation = Token Embeddings + Segment Embeddings + Position Embeddings
- WordPiece tokenization, 30K vocab
[CLS]token at start → used for classification tasks[SEP]token between sentence pairs
Pre-training tasks
Task 1: Masked Language Model (MLM)
- Randomly mask 15% of input tokens
- Of masked positions: 80% →
[MASK], 10% → random token, 10% → unchanged - The 80/10/10 split prevents the model from only learning to fill
[MASK]tokens (which never appear at fine-tuning)
Task 2: Next Sentence Prediction (NSP)
- Binary task: given sentences A and B, predict IsNext or NotNext
- 50% of pairs are actual consecutive sentences; 50% are random pairings
- Trains understanding of inter-sentence relationships for QA and NLI
Training data: BooksCorpus (800M words) + English Wikipedia (2,500M words)
Pre-training cost: 4 days on 64 TPUs for BERT-LARGE
Fine-tuning
Add a minimal output layer on top of BERT and fine-tune all parameters end-to-end:
- Classification (sentiment, NLI): feed
[CLS]vector into linear layer - Token labeling (NER, QA span detection): apply linear layer to each token output
- Fine-tuning time: ~1 hour on a single Cloud TPU
Results
GLUE (General Language Understanding Evaluation):
| Model | Score |
|---|---|
| OpenAI GPT | 75.1 |
| BERT-BASE | 79.6 |
| BERT-LARGE | 82.1 (official: 80.5) |
SQuAD (reading comprehension):
- BERT-LARGE single model: F1 90.9 on SQuAD v1.1
- BERT-LARGE ensemble: F1 93.2 (human: 91.2)
- SQuAD v2.0 F1: 83.1 (+5.1pp over prior SOTA)
BERT surpassed SOTA on 11 NLP tasks upon release.
Why bidirectionality matters
For question answering: “What is the capital of France? … Paris is located in northern France.”
GPT (left-to-right) must predict “Paris” having only seen the question. BERT can see the full passage when encoding any position — crucial for extractive QA.
Limitations
- Pre-train/fine-tune mismatch:
[MASK]appears during pre-training but never in real text — partially addressed by the 10%/10% substitution strategy - NSP debated: RoBERTa (2019) showed removing NSP matches or improves performance
- Encoder-only: cannot directly generate text; unsuitable for seq2seq tasks as-is
- Heavy pre-training cost limits reproducibility
Influence
BERT established the pre-train → fine-tune paradigm as the dominant NLP approach and triggered a generation of variants:
- RoBERTa (2019): longer training, no NSP, dynamic masking
- ALBERT (2019): parameter sharing, sentence-order prediction
- DistilBERT (2019): knowledge distillation for smaller models
- SciBERT, BioBERT, ClinicalBERT: domain-specific pre-training
See also
- vaswani-2017-attention-is-all-you-need — Transformer architecture BERT is built on
- brown-2020-language-models-are-few-shot-learners — GPT-3: the other major paradigm (decoder, few-shot)