Attention Is All You Need (Transformer)

Paper: Attention Is All You Need
Authors: Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin (Google Brain/Research)
Year: 2017 | NIPS 2017 | arXiv: 1706.03762
Source: vaswani-2017-attention-is-all-you-need

What Is It?

Transformer — sequence transduction model built entirely on attention mechanisms, with no recurrence (RNN/LSTM) and no convolution. The foundational architecture behind GPT, BERT, and virtually all modern LLMs.

The core insight: attention can replace sequential computation entirely, allowing full parallelization during training while handling long-range dependencies better than RNNs.

Architecture

                    Output Probabilities
                          ↑
                    Linear + Softmax
                          ↑
              ┌─── Decoder (×6) ───┐
              │  Masked Self-Attn  │  ← causal mask (auto-regressive)
              │  Encoder-Decoder   │  ← queries from decoder, K/V from encoder
              │  Attn              │
              │  FFN               │
              └────────────────────┘
                          ↑ encoder output
              ┌─── Encoder (×6) ───┐
              │  Self-Attention    │  ← all positions attend to all
              │  FFN               │
              └────────────────────┘
                          ↑
            Embeddings + Positional Encoding

Base model config: N=6, d_model=512, d_ff=2048, h=8, d_k=d_v=64 → 65M params
Big model config: N=6, d_model=1024, d_ff=4096, h=16 → 213M params

Core Mechanisms

1. Scaled Dot-Product Attention

Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V

Q (Query): what we’re looking for
K (Key): what each position offers as an index
V (Value): the actual content to retrieve
√d_k scaling: prevents dot products from growing too large and saturating softmax gradients

2. Multi-Head Attention

Run h=8 attention heads in parallel, each in a lower-dimensional subspace (d_k=64):

MultiHead(Q,K,V) = Concat(head_1,...,head_8) · Wᴼ

Why multi-head? A single head averages across all positions. Multiple heads can simultaneously attend to different syntactic and semantic relationships (e.g., one head for coreference, another for subject-verb agreement).

3. Three Attention Patterns

Where	Q from	K,V from	Purpose
Encoder self-attention	encoder layer	encoder layer	each token attends to all tokens
Decoder masked self-attn	decoder layer	decoder layer	causal: can’t see future tokens
Encoder-decoder attention	decoder layer	encoder output	decoder reads the source

4. Feed-Forward Network (per position)

FFN(x) = max(0, xW₁ + b₁)W₂ + b₂

d_model=512 → d_ff=2048 → d_model=512. Same weights across positions, different per layer.

5. Positional Encoding

No recurrence = no inherent position info. Solution: add sinusoids to embeddings.

PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Key property: PE(pos+k) is a linear function of PE(pos) → model can learn relative positions. Generalizes to sequence lengths unseen during training.

Why Better Than RNN?

	Self-Attention	RNN	CNN
Sequential ops	O(1)	O(n)	O(1)
Max path length	O(1)	O(n)	O(log n)
Complexity/layer	O(n²d)	O(nd²)	O(knd²)
Parallelizable	Fully	No	Partially

RNNs must process tokens one at a time → can’t parallelize training, and long-range gradients vanish over O(n) steps. Self-attention connects any two positions in a single step.

Note: O(n²d) becomes expensive for very long sequences (n >> d), which motivated later work (Flash Attention, Longformer, etc.).

Results

WMT 2014 Translation

Model	EN→DE	EN→FR	Training FLOPs
Best prior ensemble	26.36	41.29	~10²⁰
Transformer (base)	27.3	38.1	3.3×10¹⁸
Transformer (big)	28.4	41.8	2.3×10¹⁹

Big model: +2 BLEU over all ensembles on EN-DE
EN-FR: <1/4 training cost of prior SOTA, still best single model

Training Details

Adam optimizer, β1=0.9, β2=0.98
LR warmup: linear for 4,000 steps → then decay ∝ 1/√step
Dropout 0.1 + label smoothing 0.1
Base: 8×P100, 12 hours | Big: 8×P100, 3.5 days

Ablations (Table 3 highlights)

Fewer heads → worse (single head: −0.9 BLEU vs 8 heads)
Smaller d_k → worse (compatibility harder to learn)
Bigger model consistently better
Sinusoidal ≈ learned positional encoding (nearly identical)

Limitations & What Came After

O(n²) attention: bottleneck for long sequences → Flash Attention (2022), Longformer, etc.
Fixed-length context: later addressed by sliding window attention, ALiBi, RoPE
Autoregressive decoding: still sequential at inference → speculative decoding, parallel decoding
No pre-training: this paper is task-specific; GPT/BERT showed massive gains from pre-training on unlabeled text

Legacy

Nearly every major LLM is a Transformer variant:

GPT family (decoder-only) → transformers
BERT → devlin-2019-bert-pre-training-of-deep-bidirectional
Vision Transformer (ViT) → used in maes-2026-leworldmodel-stable-end-to-end-joint-embedding-predictive as the encoder
The KV cache optimization in inference (described in kih-2026-ai-ecosystem-semiconductor-market) directly stems from this architecture’s K and V matrices

AI/ML Wiki

Explorer

vaswani-2017-attention-is-all-you-need