Learning Representations by Back-propagating Errors
Authors: David Rumelhart, Geoffrey Hinton, Ronald Williams
Year: 1986 | Nature
Source: rumelhart-1986-learning-representations-by-back-propagating-errors
What it is
This 4-page Nature paper introduced backpropagation to the world — the algorithm that makes multi-layer neural networks trainable. It solved the key open problem that Minsky and Papert (1969) had exposed: how do you adjust the weights of hidden layers when there’s no direct target signal for them? Backpropagation answered by applying the chain rule of calculus recursively from output to input.
The problem it solved
A perceptron (1 layer) can only learn linearly separable functions. To learn XOR and other non-linear functions, you need hidden layers. But how do you train hidden units?
- Output units: error = (target − output) → easy to compute a gradient
- Hidden units: what should they output? No teacher signal exists for intermediate layers
The key insight: the chain rule of calculus propagates the error signal backward through the network, layer by layer. Each hidden unit’s gradient is computed as a weighted sum of the gradients of the units it feeds into.
The algorithm
Forward pass
net_j = Σᵢ wᵢⱼ · aᵢ (weighted sum of inputs)
a_j = σ(net_j) (apply sigmoid activation)
Error at output
δ_k = (y_k − a_k) · σ'(net_k)
Backward pass (the key step)
δ_j = σ'(net_j) · Σₖ wⱼₖ · δₖ
The error for unit j = its derivative × (sum of downstream errors × weights going forward).
Weight update
Δwᵢⱼ = η · δⱼ · aᵢ
Applied layer by layer from output to input: back-propagation.
What makes it work
The chain rule: if L = f(g(x)), then dL/dx = (dL/dg) · (dg/dx).
Applied to a neural network: the gradient of the loss with respect to a weight in layer l is the product of:
- The gradient at the output (layer L)
- The derivative through each intermediate activation
- The derivative through each intermediate weight matrix
Backpropagation computes all of these efficiently in O(W) operations — the same cost as a forward pass.
The critical experiment: hidden representations
The paper’s most important claim is not the algorithm itself (Werbos had discovered it in 1974) — it’s the empirical demonstration that hidden units spontaneously discover useful representations.
Family relations experiment: A network learned a database of family relationships (X’s father is Y, etc.) across two fictional families. The hidden units, trained only on input-output pairs, spontaneously organized themselves to encode:
- Which family (English vs. Italian)
- Which generation (grandparent/parent/child)
- Which branch (maternal/paternal)
Nobody told the network about these features. They emerged from the learning pressure to predict relationships correctly. This was the empirical proof that deep networks could discover structure — not just memorize mappings.
Why 1986, not 1974
Werbos derived essentially the same algorithm in his 1974 PhD thesis. Parker (1985) independently rediscovered it. Why did the 1986 paper succeed where earlier work hadn’t?
- Nature: the journal’s reach was enormous
- Demonstration: not just an algorithm, but compelling experiments showing what the algorithm discovers
- PDP context: the paper was part of the two-volume Parallel Distributed Processing collection (Rumelhart & McClelland, 1986) — a comprehensive connectionist research program that gave the community a shared framework
- Timing: the hardware was finally fast enough to run interesting experiments
What it unlocked
| Year | Development |
|---|---|
| 1988 | TDNN (time-delay neural network) for speech |
| 1989 | LeCun applies backprop to convolutional networks |
| 1997 | LSTM: backprop through time for recurrent networks |
| 2006 | Deep belief networks (pretraining + backprop fine-tuning) |
| 2012 | AlexNet: GPU-accelerated backprop at ImageNet scale |
| 2017 | Transformer: backprop through attention |
| 2020s | LLMs: backprop through 100B+ parameters |
Every neural network today is trained with backpropagation or a close variant. The 1986 paper is the direct ancestor.
See also
- minsky-1969-perceptrons-an-introduction-to-computational — The limitation backprop overcame
- rosenblatt-1958-the-perceptron-a-probabilistic-model — Single-layer predecessor
- lecun-1998-gradient-based-learning-applied-to-document — Backprop applied to CNNs
- hochreiter-1997-long-short-term-memory — Backprop through time for sequences