Scaling Laws for Neural Language Models

Authors: Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al.
Year: 2020 | arXiv
Source: kaplan-2020-scaling-laws-for-neural-language

What it is

This paper discovered that the loss of Transformer language models follows smooth power laws with model size, dataset size, and compute — across more than 7 orders of magnitude with no signs of a ceiling. The implication was that language model performance is predictable: you can extrapolate from small runs to forecast the performance of a model 1000× larger. This turned scaling from intuition into engineering.

The three power laws

When the other two variables are not bottlenecks:

L(N) ≈ (Nc/N)^{0.076}     model size
L(D) ≈ (Dc/D)^{0.095}     dataset size
L(C) ≈ (Cc/C)^{0.050}     compute budget

Where:

N = non-embedding parameters
D = training tokens
C = total FLOPs

Each law holds across 5–7 orders of magnitude. Doubling N reduces loss by the same multiplicative factor regardless of whether N is 10M or 10B.

Key findings

Architecture shape barely matters

Holding N fixed, varying depth vs. width by a factor of 2–8× changes performance by less than a few percent. What matters is the total parameter count, not how those parameters are arranged across layers.

Large models are more compute-efficient

For a fixed compute budget C, what’s the optimal allocation between model size and training tokens?

N_optimal ∝ C^{0.73}
D_optimal ∝ C^{0.27}

As compute doubles, the optimal model grows by ~1.65× but only needs ~1.21× more data.

Consequence: the compute-optimal strategy is to train large models on relatively few tokens (stopping well before convergence), not to run small models to full convergence.

Large models learn faster

A large model reaches the same loss as a small model in far fewer gradient steps and far fewer training tokens. Larger models are sample-efficient.

Overfitting scales with N/D ratio

Performance degrades when model size is large relative to dataset size. The penalty grows with N^{0.74}/D. Rule of thumb: when the model grows 8×, the dataset needs to grow ~5× to avoid overfitting.

Practical implication: predicting GPT-3

Small run (N=1B):   L ≈ 2.3
Medium run (N=10B): L ≈ 2.15
Extrapolation:
GPT-3 (N=175B):     predicted L ≈ 1.98

The scaling laws made GPT-3’s expected performance calculable before training it. Without this analysis, committing to train a 175B-parameter model would have been a gamble. With it, the outcome was a prediction, not a hope.

The compute-optimal recipe

1. Fix compute budget C (e.g., number of GPU-hours)
2. Choose N = C^{0.73} × constant
3. Train for D = C^{0.27} × constant tokens
4. Stop before convergence (larger model, fewer steps)

This is why GPT-3 was trained on 300B tokens rather than training to convergence — the Kaplan laws said a 175B model trained that way would outperform a smaller model trained longer on the same compute.

Subsequent revision: Chinchilla

Hoffmann et al. (2022) revisited this analysis with more careful controls and found the Kaplan scaling over-parameterizes models relative to data:

Paper	Optimal N:D ratio
Kaplan 2020	Train large, use less data
Chinchilla 2022	Equal scaling: N and D should grow together

Chinchilla’s optimal formula: for a compute budget C, set N ≈ D (both scale as C^{0.5}). This means models like GPT-3 were undertrained — the same compute budget on a smaller model with more data would have given better results.

Both papers agree on the existence and smoothness of power laws; they disagree on the optimal exponent split.

What scaling laws mean for research

Before scaling laws, capability improvements came from architectural innovations (attention, residual connections, normalization). Scaling laws added a second axis: given any architecture, you can predict how much better it will get with more parameters and data, and you can choose the compute-optimal allocation.

This made “just scale it up” a principled strategy, not a brute-force one.

AI/ML Wiki

Explorer

kaplan-2020-scaling-laws-for-neural-language