AI/ML Wiki

hinton-2006-a-fast-learning-algorithm-for

4 min read

A Fast Learning Algorithm for Deep Belief Nets

Authors: Geoffrey E. Hinton, Simon Osindero, Yee-Whye Teh
Year: 2006 | Neural Computation
Source: hinton-2006-a-fast-learning-algorithm-for

What it is

The paper that restarted deep learning. Hinton, Osindero and Teh showed that a deep network could be trained effectively by greedy layer-wise pretraining: train one layer at a time as a Restricted Boltzmann Machine, then fine-tune the whole stack with backpropagation. This solved the vanishing gradient problem that had made deep networks impractical since 1986 — not by fixing the gradient itself, but by initializing the weights in a region where gradients could flow.

The problem it solved

By 2006, “deep learning” was an aspiration without a practical method. The obstacle:

  • Networks deeper than 2–3 layers couldn’t be trained reliably with backpropagation
  • Random initialization led to poor local optima in deep networks
  • Gradients from the output vanished to near-zero by the time they reached early layers
  • SVMs with handcrafted features outperformed neural networks on most benchmarks

The community’s consensus: deep networks don’t work in practice.

Greedy layer-wise pretraining

The solution: don’t train the deep network end-to-end from random initialization. Instead, build it bottom-up.

Step 1: Train an RBM on the raw input. The RBM learns a distribution over the input and produces a hidden layer h₁.

Step 2: Freeze the first RBM. Treat h₁ as data and train a new RBM → h₂.

Step 3: Repeat for each subsequent layer.

Step 4: Unroll all RBMs into a deep directed network. Fine-tune end-to-end with backpropagation.

Data → RBM₁ → h₁ → RBM₂ → h₂ → RBM₃ → h₃ → classifier
         ↑ pretrain  ↑ pretrain  ↑ pretrain
       ←────────── fine-tune (backprop) ──────────────→

Each pretraining step maximizes a lower bound on the log-likelihood of the data — so each added layer provably improves the generative model. The pretraining gives backpropagation a good starting point, not a random one.

Restricted Boltzmann Machine (RBM)

An RBM is a two-layer undirected model (visible v, hidden h) with energy:

E(v, h) = −bᵥᵀv − bₕᵀh − vᵀWh

Training uses contrastive divergence (CD): approximate the gradient by a short Gibbs chain rather than full MCMC. Fast (a few steps suffice), and effective enough for pretraining.

Results

A 3-layer deep belief net (500–500–2000 units) trained on MNIST:

  • 1.25% test error — competitive with convolutional networks
  • Better than all previous non-convolutional methods
  • The generative model learned meaningful representations: units in higher layers responded to abstract features (digit identity) rather than raw pixels

Why the approach was eventually superseded

Greedy pretraining was a workaround. Later work found better direct solutions:

  • Xavier/He initialization (2010, 2015) — proper random initialization for deep networks
  • ReLU activations (Krizhevsky 2012) — gradient flows without saturation, no pretraining needed
  • Batch normalization (2015) — stabilizes training of very deep networks
  • Residual connections (2016) — allows networks 100+ layers deep

By 2012, AlexNet was trained from random initialization with none of the RBM machinery. But the 2006 paper was what broke the wall: it demonstrated empirically that depth helps when the training problem is solved, motivating the engineering work that produced AlexNet.

Historical position

Rumelhart et al. 1986 → networks up to ~3 layers
Minsky & Papert 1969  → "deep networks won't work"

Hinton et al. 2006    → deep networks CAN work with pretraining
                                 ↓
Krizhevsky et al. 2012 → deep networks work WITHOUT pretraining, with better engineering
                                 ↓
He et al. 2016        → 152-layer residual networks (no pretraining, no special tricks)

See also

Graph View

Backlinks