Source Summary: Attention Is All You Need
Full title: Attention Is All You Need
Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin
Affiliation: Google Brain / Google Research / University of Toronto
Venue: NIPS 2017 (NeurIPS) | arXiv: 1706.03762v5 (Dec 6, 2017)
PDF: papers/vaswani-2023-attention-is-all-you-need.pdf
Contributions
- Transformer architecture — first sequence transduction model relying entirely on self-attention, eliminating recurrence and convolutions completely.
- Scaled Dot-Product Attention — simple, parallelizable attention function with √d_k scaling to prevent gradient vanishing in large dimensions.
- Multi-Head Attention — h=8 parallel attention heads in different representation subspaces, concatenated.
- Sinusoidal Positional Encoding — injects position information without learned parameters; generalizes to longer sequences than seen during training.
- New SOTA on WMT 2014 EN-DE (28.4 BLEU) and EN-FR (41.8 BLEU) at dramatically lower training cost.
Methodology
Architecture Overview
Input Embeddings + Positional Encoding
↓
[Encoder × 6]
└─ Multi-Head Self-Attention
└─ Add & Norm (residual)
└─ Feed-Forward Network (FFN)
└─ Add & Norm
↓
[Decoder × 6]
└─ Masked Multi-Head Self-Attention
└─ Add & Norm
└─ Multi-Head Encoder-Decoder Attention
└─ Add & Norm
└─ Feed-Forward Network
└─ Add & Norm
↓
Linear + Softmax → Output Probabilities
Hyperparameters (base model):
- N = 6 layers, d_model = 512, d_ff = 2048
- h = 8 heads, d_k = d_v = 64
- Dropout P_drop = 0.1, label smoothing ε_ls = 0.1
- 65M parameters
Big model: N=6, d_model=1024, d_ff=4096, h=16, P_drop=0.3 → 213M parameters
Scaled Dot-Product Attention
Attention(Q, K, V) = softmax(QK^T / √d_k) · V
- Scaling by √d_k prevents dot products from growing large and pushing softmax into low-gradient regions.
- Implemented as matrix multiplication → highly optimized on hardware.
Multi-Head Attention
MultiHead(Q, K, V) = Concat(head_1, ..., head_h) · W^O
where head_i = Attention(QW^Q_i, KW^K_i, VW^V_i)
- h=8 heads, each projecting to d_k=d_v=64
- Total computation ≈ single-head attention at full d_model
- Allows attending to different representation subspaces simultaneously
Three Uses of Attention in Transformer
- Encoder self-attention: each position attends to all positions in previous encoder layer
- Decoder masked self-attention: each position attends only to positions ≤ current (causal masking via −∞)
- Encoder-decoder attention: decoder queries attend to all encoder output positions
Position-wise FFN
FFN(x) = max(0, xW_1 + b_1) · W_2 + b_2
- d_model=512 in/out, d_ff=2048 inner
- Same transformation across positions, different parameters per layer
Positional Encoding
PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
- Added to input embeddings; same dimension d_model
- Sinusoidal chosen over learned embeddings: can extrapolate to unseen sequence lengths
- Nearly identical results between sinusoidal and learned (Table 3 row E)
Training
- Optimizer: Adam (β1=0.9, β2=0.98, ε=10^-9)
- LR schedule: linear warmup for 4,000 steps, then decay ∝ 1/√step
- Regularization: residual dropout (0.1), label smoothing (0.1)
- Hardware: 8 NVIDIA P100 GPUs; base model 12h, big model 3.5 days
Results
Machine Translation (WMT 2014)
| Model | EN-DE BLEU | EN-FR BLEU | Training Cost |
|---|---|---|---|
| ConvS2S Ensemble | 26.36 | 41.29 | 7.7×10¹⁹ FLOPs |
| GNMT+RL Ensemble | 26.30 | 41.16 | 1.8×10²⁰ FLOPs |
| Transformer (base) | 27.3 | 38.1 | 3.3×10¹⁸ FLOPs |
| Transformer (big) | 28.4 | 41.8 | 2.3×10¹⁹ FLOPs |
- Big model surpasses all ensembles on EN-DE by >2 BLEU
- EN-FR big model: less than 1/4 training cost of prior SOTA
Why Self-Attention vs. RNN/CNN (Table 1)
| Layer Type | Complexity/Layer | Sequential Ops | Max Path Length |
|---|---|---|---|
| Self-Attention | O(n²·d) | O(1) | O(1) |
| Recurrent | O(n·d²) | O(n) | O(n) |
| Convolutional | O(k·n·d²) | O(1) | O(log_k(n)) |
Key insight: self-attention connects all positions in O(1) sequential steps, enabling better long-range dependency learning. n < d in typical NLP tasks, so O(n²·d) < O(n·d²).
English Constituency Parsing
- 4-layer Transformer, d_model=1024
- Semi-supervised: 92.7 F1 (WSJ Sec.23) — competitive with task-specific models
- Shows generalization beyond translation
Limitations
- O(n²) attention complexity becomes costly for very long sequences
- Authors note restricted self-attention (neighborhood of size r) as future work → later addressed by Longformer, BigBird, Flash Attention, etc.
- Autoregressive decoding remains sequential at inference time
Key References
- Bahdanau et al. 2014 — additive attention (precursor)
- Hochreiter & Schmidhuber 1997 — LSTM
- He et al. 2016 — residual connections
- Ba et al. 2016 — layer normalization
- Kingma & Ba 2015 — Adam optimizer
- Shazeer et al. 2017 — Mixture of Experts (baseline)