ImageNet Classification with Deep Convolutional Neural Networks
Authors: Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton
Year: 2012 | NeurIPS
Source: krizhevsky-2012-imagenet-classification-with-deep-convolutional
What it is
AlexNet won ILSVRC-2012 with 15.3% top-5 error against a second-place score of 26.2% — a 10.9 percentage-point gap that ended the era of hand-crafted features in computer vision. The paper demonstrated that the CNN architecture LeCun had designed in 1998 could be scaled to 1.2 million images using GPUs, and that doing so produced a step-change in capability.
Architecture
Input: 224×224×3 RGB image
Conv1: 96 kernels, 11×11, stride 4 → 55×55×96 + MaxPool
Conv2: 256 kernels, 5×5, pad 2 → 27×27×256 + MaxPool
Conv3: 384 kernels, 3×3, pad 1 → 13×13×384
Conv4: 384 kernels, 3×3, pad 1 → 13×13×384
Conv5: 256 kernels, 3×3, pad 1 → 13×13×256 + MaxPool
FC6: 4096 units (Dropout 0.5)
FC7: 4096 units (Dropout 0.5)
FC8: 1000 units → Softmax
~60M parameters. Training: 5–6 days on 2 GTX 580 GPUs (3GB VRAM each).
Four key innovations
1. ReLU activations
Prior networks used tanh or sigmoid, which saturate for large inputs — the gradient approaches zero and learning stalls. ReLU is:
f(x) = max(0, x)
Properties:
- No saturation for positive inputs: gradient is always 1 (or 0 for dead neurons)
- Computationally cheap: no exponentiation
- Sparse: ~half of neurons output 0 at any time
On CIFAR-10, a 4-layer ReLU network reached 25% training error 6× faster than the equivalent tanh network.
2. GPU training
The network was split across two GTX 580 GPUs. Each GPU held half the kernels; they communicated only at specific layers. Custom CUDA convolution kernels made training practical.
Without GPUs, training a 60M-parameter network on 1.2M images would have taken months on contemporary CPUs.
3. Dropout
During training, each unit in the FC layers is independently set to 0 with probability 0.5. At test time, all units are active but outputs are scaled by 0.5.
Effect: prevents co-adaptation — a unit can’t rely on specific other units being present, so it’s forced to learn independently useful features. Equivalent to training an ensemble of ~2^N different networks with shared weights.
Without dropout, AlexNet overfits substantially. With it, the train/val gap narrows dramatically.
4. Data augmentation
- Random 224×224 crops from 256×256 images
- Horizontal reflections
- PCA color jitter: add scaled principal components of RGB values with Gaussian-sampled magnitudes
Effective training set grows by a factor of ~2048 without collecting new data.
Results
| Method | Top-5 error |
|---|---|
| Best non-CNN method (2011 winner) | 25.8% |
| ILSVRC-2012 runner-up (non-CNN) | 26.2% |
| AlexNet (1 model) | 16.4% |
| AlexNet (ensemble of 5) | 15.3% |
| Human | ~5% |
The runner-up used SIFT + Fisher vectors — hand-crafted features that had dominated computer vision for a decade. The margin wasn’t marginal; it was a different category of performance.
Why this paper changed everything
Before 2012, the dominant paradigm was:
- Design hand-crafted features (SIFT, HOG, SURF, Fisher vectors)
- Train a classifier (SVM, random forest) on those features
- Win competitions by better feature engineering
After AlexNet:
- Every ILSVRC winner from 2013 onward used deep CNNs
- Computer vision research shifted entirely to end-to-end deep learning
- The same pattern propagated to speech (had already shifted in 2011), NLP, and eventually every domain
The three preconditions were: deep architecture (5 conv layers, proven by LeCun 1998) + GPU compute (GTX 580, 2010) + large labeled data (ImageNet, 2009). All had existed in some form — AlexNet combined them at the right scale.
See also
- lecun-1998-gradient-based-learning-applied-to-document — LeNet-5: the CNN architecture being scaled
- rumelhart-1986-learning-representations-by-back-propagating-errors — Backpropagation: the training algorithm
- vaswani-2017-attention-is-all-you-need — Transformer: the architecture that succeeded CNNs for NLP