Source Summary: Rumelhart, Hinton & Williams 1986 — Backpropagation
Full title: Learning Representations by Back-propagating Errors
Authors: David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams
Affiliation: University of California San Diego / Carnegie Mellon University
Venue: Nature, Vol. 323, pp. 533–536 (9 October 1986)
PDF: papers/rumelhart-1986-learning-representations-by-back-propagating-errors.pdf
Note: PDF is a scanned Nature reprint; content based on the paper itself.
Contributions
- Backpropagation algorithm — efficient computation of the gradient of a loss function with respect to all weights in a multi-layer neural network via the chain rule of calculus.
- Hidden layer representations — demonstrated empirically that hidden units learn meaningful internal representations of the input, not just output mappings.
- Overcame the XOR problem — multi-layer networks trained with backpropagation can solve problems that single-layer perceptrons cannot (including XOR), answering Minsky & Papert’s 1969 critique.
- Generality — the algorithm applies to any differentiable network regardless of depth or topology.
The Algorithm
Forward pass
For a network with layers l = 1, …, L:
aˡ = σ(Wˡ aˡ⁻¹ + bˡ)
where aˡ is the activation vector of layer l, Wˡ is the weight matrix, bˡ is the bias, and σ is the activation function (e.g., sigmoid).
Loss function
E = ½ Σₖ (yₖ − aᵏₗ)² (sum-of-squares error)
where yₖ is the target and aᵏₗ is the output unit activation.
Backward pass
Define error signal (delta) for the output layer:
δˡₖ = (yₖ − aˡₖ) · σ'(netˡₖ)
For hidden layer l (working backwards):
δˡⱼ = σ'(netˡⱼ) · Σₖ wˡ⁺¹ₖⱼ δˡ⁺¹ₖ
The error is propagated backward through the network using the transposed weight matrix.
Weight update
ΔWˡᵢⱼ = η · δˡᵢ · aˡ⁻¹ⱼ
where η is the learning rate.
The key insight: the chain rule of calculus allows the gradient to be computed layer by layer, efficiently. The computation cost is O(W) per training example, where W is the number of weights — the same order as the forward pass.
Key Experiments
XOR
A two-layer network (2 input, 2 hidden, 1 output) successfully learned XOR after training — the canonical problem that Minsky & Papert proved single-layer perceptrons cannot solve.
Symmetry detection
A 6-input network learned to detect whether a binary pattern is symmetric, discovering an internal representation that captures mirror symmetry.
Family relations task
A network with multiple hidden layers learned to represent family relationships (e.g., “X’s father is Y” across two families). Hidden units spontaneously developed representations of nationality, generation, and branch of the family — features never explicitly provided.
The Key Insight: Hidden Representations
The paper’s central empirical claim: when trained with backpropagation, hidden units do not just learn input-output mappings. They spontaneously develop useful internal representations of the input. The network discovers features.
This was the answer to the major open question in connectionism after Minsky & Papert: not only can multi-layer networks represent more functions, but they can be trained to do so.
Historical Note
The backpropagation algorithm itself was discovered independently multiple times:
- Linnainmaa 1970 (automatic differentiation)
- Werbos 1974 (PhD thesis)
- Parker 1985
- Rumelhart, Hinton & Williams 1986 — the paper that made the field take notice
The 1986 paper succeeded where earlier work hadn’t because: (1) published in Nature, (2) combined with compelling experimental demonstrations, (3) was part of the PDP (Parallel Distributed Processing) research program with broad visibility.
Key References
- Rosenblatt 1958 — single-layer perceptron (the limitation being overcome)
- Minsky & Papert 1969 — the critique backpropagation answered
- Werbos 1974 — independent earlier discovery of backpropagation
- LeCun et al. 1989/1998 — applied backpropagation to convolutional networks