Source Summary: Kaplan et al. 2020 — Scaling Laws
Full title: Scaling Laws for Neural Language Models
Authors: Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei
Affiliation: Johns Hopkins University / OpenAI
Venue: arXiv: 2001.08361v1 (January 23, 2020)
PDF: papers/kaplan-2020-scaling-laws-for-neural-language.pdf
Contributions
- Power-law scaling with model size: L(N) ∝ N^{−α_N}, α_N ≈ 0.076 — loss decreases smoothly as a power law with the number of non-embedding parameters N, across 5+ orders of magnitude.
- Power-law scaling with data: L(D) ∝ D^{−α_D}, α_D ≈ 0.095 — same relationship with dataset size D.
- Power-law scaling with compute: L(C) ∝ C^{−α_C}, α_C ≈ 0.050 — and with compute budget C.
- Compute-optimal training: for a fixed compute budget, optimal to train a larger model on fewer tokens (stopping short of convergence) rather than a smaller model to full convergence.
- Architecture details matter little: within a wide range, depth vs. width has minimal effect; what matters is total parameter count.
Methodology
Models trained
- Transformer language models (decoder-only)
- Sizes: 768M to 1.5B parameters (and smaller models for scaling)
- Trained on WebText2 (a cleaned web corpus)
- Evaluated by cross-entropy loss on held-out text
Scaling axes studied
- N = number of non-embedding parameters
- D = dataset size in tokens
- C = total compute (FLOPs)
- Context length, batch size, depth, width also varied
Key Findings
Finding 1: Smooth power laws
Loss follows a power law with each variable when the other two are not bottlenecks:
L(N) ≈ (Nc/N)^{αN} αN ≈ 0.076
L(D) ≈ (Dc/D)^{αD} αD ≈ 0.095
L(C) ≈ (Cc/C)^{αC} αC ≈ 0.050
The same curves hold across more than 7 orders of magnitude. No signs of a floor or ceiling within the studied range.
Finding 2: Architecture shape doesn’t matter much
Within a factor of 2–8× in depth vs. width (holding N fixed), performance varies by less than a few percent. The total parameter count is what matters, not how those parameters are arranged.
Finding 3: Compute-efficient training
For a fixed compute budget C, there is an optimal model size N*(C) and an optimal number of training tokens D*(C).
Key result: larger models are more compute-efficient.
Training a 10× larger model for 10× fewer steps typically gives better loss than a smaller model trained to convergence.
This implies:
N_optimal ∝ C^{0.73}
D_optimal ∝ C^{0.27}
So as compute doubles, the optimal model grows by 1.65× but only needs 1.21× more data.
Finding 4: Sample efficiency of large models
Large models reach the same loss as small models with far fewer gradient updates and far fewer training tokens. The curves intersect — large models start slow but finish faster.
Finding 5: Overfitting depends on ratio N/D
Performance degrades when the model size N is large relative to the dataset D. The penalty depends on:
N^{0.74} / D
Rule of thumb: when the model grows 8×, the dataset needs to grow ~5× to avoid a penalty.
Implications for GPT-3
This paper directly motivated GPT-3’s design:
- Train a very large model (175B params)
- On relatively moderate data (300B tokens)
- Stop before convergence
Without the scaling laws analysis, the GPT-3 decision to train at 175B would have been a gamble. With it, the expected performance was predictable from the scaling curves.
Summary Table of Power Laws
| Variable | Exponent | Implication |
|---|---|---|
| Parameters N | -0.076 | 10× more params → 40% lower loss |
| Data D | -0.095 | 10× more data → 45% lower loss |
| Compute C | -0.050 | 10× more compute → 30% lower loss |
| Context length | varies | Diminishing returns after moderate length |
Limitations
- Results specific to Transformer language models on web text; may not generalize to all architectures or modalities
- The Chinchilla paper (Hoffmann et al. 2022) revised the optimal compute-allocation formula: original Kaplan scaling over-parameterizes models relative to data; Chinchilla says equal scaling of N and D is optimal
- Does not address sample quality, only loss — does lower loss always mean better capabilities?
Key References
- Brown et al. 2020 (GPT-3) — applied these scaling laws
- Hoffmann et al. 2022 (Chinchilla) — revised the compute-optimal formula
- Vaswani et al. 2017 — Transformer architecture being studied