Source Summary: LeCun et al. 1998 — LeNet / CNNs
Full title: Gradient-Based Learning Applied to Document Recognition
Authors: Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner
Affiliation: AT&T Bell Labs / AT&T Labs-Research
Venue: Proceedings of the IEEE, Vol. 86, No. 11, pp. 2278–2324 (November 1998)
PDF: papers/lecun-1998-gradient-based-learning-applied-to-document.pdf
Contributions
- LeNet-5 — a 7-layer convolutional neural network that achieves <1% error on MNIST handwritten digits; the canonical CNN architecture.
- Convolutional layers — locally connected, weight-sharing layers that exploit spatial structure; dramatically fewer parameters than fully connected networks.
- Gradient-based training for document recognition — end-to-end backpropagation through convolution, pooling, and fully connected layers.
- Graph Transformer Networks (GTN) — a framework for building end-to-end trainable systems that include heuristic components (segmentation, word models).
- Systematic comparison — benchmarks LeNet-5 against boosted decision trees, SVMs, nearest neighbors, and other methods; CNN wins on both accuracy and speed.
LeNet-5 Architecture
Input (32×32)
→ C1: 6 feature maps, 5×5 kernels → 28×28×6
→ S2: Average pooling 2×2 → 14×14×6
→ C3: 16 feature maps, 5×5 kernels → 10×10×16
→ S4: Average pooling 2×2 → 5×5×16
→ C5: 120 units, 5×5 kernels (FC) → 120
→ F6: Fully connected, 84 units → 84
→ Output: 10 classes (RBF units) → 10
Total parameters: ~60,000 (vs. millions for a fully connected network on same task)
Convolutional layer
Each unit in a feature map receives input from a 5×5 receptive field in the previous layer. All units in a feature map share the same weights — the same filter is applied at every position.
This implements:
- Local connectivity: each unit sees only a small region (invariant to exact position)
- Weight sharing: the same feature detector is applied everywhere (translation equivariance)
- Hierarchical features: early layers detect edges, later layers detect shapes
Pooling layer (subsampling)
Each 2×2 region maps to a single output (average pooling). Provides mild translation invariance — the network is somewhat insensitive to small shifts in the input.
Why CNNs beat fully connected networks
| Property | FC network | CNN |
|---|---|---|
| Parameters for 32×32 input | ~O(input × output) | ~O(filter × maps) |
| Translation invariance | None | Built-in via weight sharing |
| Exploits spatial structure | No | Yes |
| Computation | O(N²) in image size | O(N²) but with much smaller constants |
Key Results (1998)
MNIST handwritten digits
| Method | Test error |
|---|---|
| Linear classifier | 12.0% |
| Fully connected NN (300 hidden) | 4.7% |
| SVM with RBF kernel | 1.4% |
| LeNet-5 | 0.95% |
| Human | ~0.2% |
Check reading
Deployed at AT&T for reading handwritten checks — at peak, reading ~10% of all checks in the US. First large-scale commercial deployment of a trained neural network.
Theoretical Framework
The paper introduces the concept of a Graph Transformer Network (GTN) — a framework for combining trainable modules (neural nets) with non-differentiable components (string decoders, HMMs) into a single end-to-end system.
The key innovation: use the Viterbi algorithm as a decoder that can be combined with a gradient-learned feature extractor. This allows joint optimization of the full pipeline (segmentation → recognition → language model) rather than training each component separately.
Comparison to Boosteing and SVMs
In 1998, SVMs and boosted classifiers were considered state-of-the-art for classification. The paper shows that:
- LeNet-5 outperforms SVMs on MNIST (0.95% vs 1.4%)
- LeNet-5 vastly outperforms SVMs on the more complex document recognition tasks where the input structure matters
- CNNs are more efficient to apply at test time than SVMs for image tasks
Historical Impact
LeNet-5 was the first practical demonstration that a deep CNN trained end-to-end with backpropagation could match or beat hand-engineered systems on a real-world problem. But adoption was slow — in 1998, computers weren’t fast enough to scale CNNs to ImageNet-scale problems, and SVMs dominated until AlexNet (2012) proved the same ideas at larger scale with GPUs.
Key References
- LeCun et al. 1989 — first application of backprop to convnets (zip code recognition)
- Rumelhart et al. 1986 — backpropagation algorithm
- Krizhevsky et al. 2012 — scaled LeNet ideas to ImageNet with GPUs