Source Summary: Hinton, Osindero & Teh 2006 — Deep Belief Nets
Full title: A Fast Learning Algorithm for Deep Belief Nets
Authors: Geoffrey E. Hinton, Simon Osindero, Yee-Whye Teh
Affiliation: University of Toronto
Venue: Neural Computation, 18(7), pp. 1527–1554 (2006)
PDF: papers/hinton-2006-a-fast-learning-algorithm-for.pdf
Contributions
- Greedy layer-wise pretraining — a fast, unsupervised algorithm that trains a deep network one layer at a time using Restricted Boltzmann Machines (RBMs); solved the vanishing gradient problem that had made deep networks untrainable since 1986.
- Deep Belief Network (DBN) — a deep generative model composed of stacked RBMs; the first deep architecture to match and surpass discriminative methods on MNIST.
- Proof of improvement with depth — shows theoretically that adding each RBM layer improves a lower bound on the log-likelihood of the training data.
- Reignited deep learning research — this paper, along with Bengio et al. 2006 and LeCun’s work, triggered the deep learning revival that culminated in AlexNet 2012.
The Problem
Deep neural networks trained end-to-end with backpropagation had a severe limitation: the vanishing gradient. In networks with many layers, gradients diminish exponentially as they propagate backward, so lower layers learn extremely slowly or not at all.
By 2006, practical neural networks were rarely deeper than 2–3 layers. SVMs and other shallow methods dominated.
The Solution: Greedy Layer-Wise Pretraining
Step 1: Train an RBM on the raw input data. The RBM learns a probability distribution over the input and produces a hidden representation h₁.
Step 2: Treat h₁ as data and train a new RBM on h₁, producing h₂.
Step 3: Repeat for each layer. Each RBM trains greedily, independently of the others.
Step 4: Unroll the stacked RBMs into a deep directed network (Deep Belief Net). Fine-tune the entire network using backpropagation (contrastive wake-sleep algorithm).
Input → [RBM₁] → h₁ → [RBM₂] → h₂ → [RBM₃] → h₃ → labels
(pretrain) (pretrain) (pretrain)
←─────────── fine-tune (backprop) ──────────────
The key insight: pretraining initializes the weights in a region where backpropagation can fine-tune effectively, avoiding the poor local optima that plagued random initialization of deep networks.
Restricted Boltzmann Machine (RBM)
An RBM is a two-layer undirected graphical model:
- Visible units v: the observed data
- Hidden units h: the learned representation
Energy function:
E(v, h) = −bᵥᵀv − bₕᵀh − vᵀWh
Training (contrastive divergence): maximize P(v) = Σₕ exp(−E(v,h)) / Z using an efficient approximation to the gradient.
Results
The deep belief network (3 hidden layers: 500, 500, 2000 units) achieves 1.25% error on MNIST — better than all previous methods except convolutional networks, using standard (non-convolutional) architecture.
More importantly: it was the first time a deep (>2 hidden layer) network showed that depth helps — that the additional layers were capturing real structure, not overfitting.
Why This Paper Changed Everything
Before 2006: deep networks = untrainable. The conventional wisdom was that more than 2–3 layers added nothing but instability.
After 2006: deep networks = trainable given proper initialization. The pretraining trick worked, and within two years, deep architectures were competing with SVMs on standard benchmarks.
The RBM pretraining approach was largely superseded by:
- Better random initialization (Glorot & Bengio 2010: Xavier initialization)
- ReLU activations (Krizhevsky et al. 2012)
- Batch normalization (Ioffe & Szegedy 2015)
- Residual connections (He et al. 2016)
But the 2006 paper broke the mental barrier: deep networks can work. AlexNet (2012) just needed the right engineering.
Key References (from paper)
- Rumelhart et al. 1986 — backpropagation (the algorithm being fixed here)
- Hinton & Salakhutdinov 2006 — concurrent paper: deep autoencoders reduce dimensionality
- LeCun et al. 1998 — LeNet (showing depth works for CNNs, ignored for fully connected nets)