Gradient-Based Learning Applied to Document Recognition
Authors: Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner
Year: 1998 | Proceedings of the IEEE
Source: lecun-1998-gradient-based-learning-applied-to-document
What it is
This paper introduced LeNet-5 — a 7-layer convolutional neural network that achieved 0.95% error on MNIST, beating SVMs and every prior method. More broadly, it demonstrated that backpropagation could train deep architectures end-to-end on real perceptual tasks, not just toy problems. At peak deployment, LeNet was reading ~10% of all checks processed in the US.
The architecture: LeNet-5
Input (32×32)
→ C1: 6 feature maps, 5×5 kernels → 28×28×6
→ S2: Average pooling 2×2 → 14×14×6
→ C3: 16 feature maps, 5×5 kernels → 10×10×16
→ S4: Average pooling 2×2 → 5×5×16
→ C5: 120 units, fully connected → 120
→ F6: Fully connected, 84 units → 84
→ Output: 10 classes (RBF units) → 10
~60,000 parameters — versus millions for a fully connected network on the same task.
The two core ideas
1. Local connectivity
Each unit in a feature map connects to only a small 5×5 receptive field in the previous layer. This exploits the spatial structure of images: nearby pixels are more correlated than distant ones. A fully connected layer ignores this structure entirely.
2. Weight sharing
All units in a feature map share the same weights — the same filter is applied at every position. This implements a convolutional operation:
output(i,j) = Σ_{m,n} w(m,n) · input(i+m, j+n) + b
Benefits:
- Parameters scale with filter size, not image size
- The same feature detector (edge, curve, corner) is learned once and applied everywhere
- Translation equivariance: a feature detected at position (3,5) also gets detected at (7,9)
Pooling
Each 2×2 region maps to a single output. Provides mild translation invariance — the network becomes somewhat insensitive to small shifts in the input position.
Why CNNs beat fully connected networks
| Property | FC network | CNN |
|---|---|---|
| Parameters for 32×32 input | ~O(input × hidden) | ~O(filter × maps) |
| Translation invariance | None | Built-in via weight sharing |
| Exploits spatial structure | No | Yes |
| Generalizes to new positions | No | Yes |
Results
MNIST (1998)
| Method | Test error |
|---|---|
| Linear classifier | 12.0% |
| Fully connected NN (300 hidden) | 4.7% |
| SVM with RBF kernel | 1.4% |
| LeNet-5 | 0.95% |
| Human | ~0.2% |
LeNet-5 outperformed SVMs despite SVMs being the dominant method of the era.
Commercial deployment
AT&T deployed LeNet-5 to read handwritten checks — the first large-scale commercial neural network. At peak, it was processing ~10% of all US check transactions.
Beyond digit recognition: Graph Transformer Networks
The paper’s larger contribution was a framework for building end-to-end trainable systems that mix:
- Neural networks (differentiable, trained by backprop)
- Non-differentiable components (segmenters, language models, Viterbi decoders)
The key: use the Viterbi algorithm as a decoder that can be combined with a learned feature extractor, allowing joint optimization of the full pipeline rather than training each module separately.
This prefigures the end-to-end learning philosophy that deep learning would make universal after 2012.
Why it didn’t immediately dominate
LeNet-5 was published in 1998. The deep learning revolution it anticipated didn’t arrive until AlexNet in 2012 — a 14-year gap. The reasons:
- Compute: training a CNN on ImageNet-scale data required GPU acceleration, unavailable in 1998
- Data: ImageNet (1.2M labeled images) didn’t exist until 2009
- Competition: SVMs had clean theoretical foundations and matched CNNs on the small datasets available
When GPUs and large datasets arrived, the same architecture LeCun had designed in 1998 turned out to work at orders-of-magnitude larger scale.
See also
- rumelhart-1986-learning-representations-by-back-propagating-errors — Backpropagation: the training algorithm
- krizhevsky-2012-imagenet-classification-with-deep-convolutional — AlexNet: the same ideas at GPU scale