Source Summary: Krizhevsky, Sutskever & Hinton 2012 — AlexNet
Full title: ImageNet Classification with Deep Convolutional Neural Networks
Authors: Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton
Affiliation: University of Toronto
Venue: Advances in Neural Information Processing Systems (NeurIPS) 25, pp. 1097–1105 (2012)
PDF: papers/krizhevsky-2012-imagenet-classification-with-deep-convolutional.pdf
Note: PDF is mostly image-based; content from the paper itself.
Contributions
- AlexNet — 8-layer deep CNN (5 conv + 3 FC) that won ILSVRC-2012 with 15.3% top-5 error vs. 26.2% for the runner-up; a 10.9pp margin.
- GPU training — distributed training across two GTX 580 3GB GPUs; made training 60M-parameter network feasible.
- ReLU activation — Rectified Linear Units (f(x) = max(0,x)) replace tanh/sigmoid; train 6× faster on CIFAR-10 with same accuracy.
- Dropout regularization — randomly zero out hidden units during training; prevents co-adaptation and reduces overfitting dramatically.
- Data augmentation — random crops, horizontal flips, PCA color jitter; artificially multiplies training set size.
Architecture (AlexNet)
Input: 224×224×3 RGB image
Conv1: 96 kernels, 11×11, stride 4 → 55×55×96
MaxPool: 3×3, stride 2 → 27×27×96
Conv2: 256 kernels, 5×5, pad 2 → 27×27×256
MaxPool: 3×3, stride 2 → 13×13×256
Conv3: 384 kernels, 3×3, pad 1 → 13×13×384
Conv4: 384 kernels, 3×3, pad 1 → 13×13×384
Conv5: 256 kernels, 3×3, pad 1 → 13×13×256
MaxPool: 3×3, stride 2 → 6×6×256
FC6: 4096 units (Dropout 0.5)
FC7: 4096 units (Dropout 0.5)
FC8: 1000 units (Softmax)
Total parameters: ~60M
Training time: 5–6 days on 2 GPUs
Key Technical Innovations
ReLU activations
Prior networks used tanh or sigmoid. Problem: these saturate — for large inputs, gradient ≈ 0, learning stalls.
ReLU (f(x) = max(0,x)):
- No saturation for positive inputs → gradient always flows
- Computationally cheaper than exp()
- Sparsifies activations (half the neurons are off at any time)
Result: same accuracy as tanh networks in 6× fewer training epochs on CIFAR-10.
GPU implementation
The network was split across two GTX 580 GPUs (3GB VRAM each):
- Each GPU holds half the kernels
- GPUs communicate only at specific layers (Layer 3 connects to all Layer 2 kernels on both GPUs)
- Custom CUDA kernels for fast convolution
This engineering allowed training a 60M parameter network in ~6 days — previously infeasible.
Dropout
During training, each hidden unit is independently zeroed with probability 0.5. At test time, all units active but outputs scaled by 0.5.
Effect: prevents units from co-adapting (one unit learning to correct another’s mistakes). Forces each unit to learn useful features independently. Equivalent to training an ensemble of ~2^N different networks.
Without dropout: AlexNet overfits substantially. With dropout: train/val gap closes dramatically.
Data augmentation
- Random 224×224 crops from 256×256 images
- Horizontal reflection
- PCA on RGB pixels: add multiples of principal components with Gaussian-sampled magnitudes (captures illumination variation)
Increases effective training set size by ~2048× without new data collection.
Local Response Normalization (LRN)
Lateral inhibition across feature maps at the same spatial location. Less important than ReLU/Dropout; largely abandoned in subsequent work.
Results (ILSVRC-2012)
| Method | Top-5 error |
|---|---|
| Best non-CNN (2011 winner) | 25.8% |
| ILSVRC-2012 2nd place | 26.2% |
| AlexNet (ensemble) | 15.3% |
| Human | ~5% |
The gap was unprecedented. The second-place system used hand-crafted SIFT/HOG features — the kind of feature engineering that had dominated computer vision for a decade.
Why It Changed Everything
Before AlexNet (2012): deep neural networks were considered too slow to train, prone to overfitting, and inferior to SVMs + hand-crafted features on most vision tasks.
After AlexNet:
- Every subsequent ILSVRC winner used deep CNNs
- Computer vision research shifted entirely to deep learning
- The same pattern repeated in speech (already happened 2011 with DNN-HMM), NLP, and eventually everything
The three enabling factors: deep architecture (5 conv layers) + GPU compute + large labeled dataset (ImageNet, 1.2M images). All three had existed in some form before 2012 — AlexNet combined them at the right scale.
Key References
- LeCun et al. 1998 — LeNet (the CNN architecture being scaled)
- Rumelhart et al. 1986 — backpropagation
- Deng et al. 2009 — ImageNet dataset
- Hinton et al. 2012 — DNN-HMM for speech (parallel deep learning breakthrough)