An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Contributions

• Pure Transformer for image classification: demonstrates that a standard Transformer encoder, applied directly to sequences of image patches, achieves state-of-the-art accuracy on image classification with no convolutional inductive biases.
• Patch-based image tokenization: introduces the idea of splitting an image into fixed-size 16×16 patches, linearly projecting each into an embedding, and treating the sequence of embeddings as tokens — making images compatible with the NLP Transformer architecture.
• Pre-training at scale enables transfer: shows that when pre-trained on large datasets (JFT-300M or ImageNet-21k), ViT matches or exceeds ResNets of comparable compute on downstream benchmarks while training faster on TPUs.
• Inductive bias analysis: establishes that CNNs embed spatial locality and translation equivariance by design, whereas ViT must learn these from data — which is a liability at small scale but becomes irrelevant with sufficient data.

Methodology

Image as Patch Sequence

Given an image x ∈ ℝ^(H×W×C), it is divided into N patches of size P×P, yielding N = HW/P² patches. Each patch is flattened to a vector of dimension P²C and projected to a D-dimensional embedding via a learned linear layer. Standard 1D learnable position embeddings are added.

A learnable [CLS] token is prepended to the sequence (following BERT). Its representation at the final layer is used as the image representation for classification via an MLP head.

Architecture

The ViT encoder is a standard Transformer: alternating layers of Multi-Head Self-Attention (MSA) and MLP blocks, with LayerNorm before each and residual connections. No convolutions, no pooling, no local receptive fields. Three model sizes are studied:

• ViT-Base: 12 layers, 768 hidden dim, 12 heads — 86M parameters
• ViT-Large: 24 layers, 1024 hidden dim, 16 heads — 307M parameters
• ViT-Huge: 32 layers, 1280 hidden dim, 16 heads — 632M parameters

Pre-training and Fine-tuning

Models are pre-trained on JFT-300M (Google internal, ~300M images) or ImageNet-21k (~14M images). Fine-tuning replaces the pre-training classification head with a task-specific head and uses higher resolution (resizing position embeddings via 2D interpolation).

A hybrid variant replaces raw patch embeddings with feature maps from a ResNet, feeding the resulting grid into the Transformer.

Results

• ImageNet (fine-tuned from JFT-300M): ViT-H/14 achieves 88.55% top-1, surpassing the best prior BiT ResNet (87.54%) and EfficientNet variants.
• ImageNet-21k pre-training: ViT-L/16 reaches 87.76% on ImageNet top-1, competitive with the best CNNs.
• Transfer tasks (CIFAR-10, CIFAR-100, Oxford Pets, Oxford Flowers): ViT-L/16 achieves near-perfect scores (99.7% on CIFAR-10, 97.8% on CIFAR-100).
• Compute efficiency: ViT-L/16 pre-trained on JFT achieves BiT’s top accuracy at ~2–4× lower pre-training compute (TPU-core-days).
• Small data regime: on ImageNet alone (no extra data), ViT-L/16 reaches only 76.5%, substantially below ResNet-50 (77.5%), confirming the need for large pre-training data.

Limitations

• Requires significantly more data than CNNs to reach comparable accuracy — the lack of convolutional inductive biases is a liability at small scale.
• Pre-training on large proprietary datasets (JFT-300M) may not be reproducible by the research community.
• Global self-attention is O(N²) in sequence length; at 16×16 patch size with 224×224 images, N=196 is manageable, but higher-resolution or smaller-patch variants are expensive.
• No explicit treatment of multi-scale features; subsequent work (DeiT, Swin Transformer) addressed this limitation.