AI/ML Wiki

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

4 min read

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Authors: Dosovitskiy, Beyer, Kolesnikov, et al. (Google Brain) Year: 2020

The Vision Transformer (ViT) is a landmark paper demonstrating that the standard Transformer architecture — designed for NLP — can match and surpass convolutional neural networks on image classification when pre-trained on sufficiently large datasets, with no convolutional operations whatsoever. The paper’s central insight is deceptively simple: treat an image as a sequence of 16×16 pixel patches, project each patch into an embedding vector, and feed the resulting sequence through an ordinary Transformer encoder. This reframing allowed the entire NLP Transformer toolbox — multi-head attention, pre-training, fine-tuning — to be applied directly to vision.

Core Idea

An image of size H×W is divided into N = HW/P² non-overlapping patches of size P×P pixels (P=16 in the standard variant). Each patch is flattened into a vector and linearly projected to dimension D, giving a sequence of N patch embeddings. A learnable [CLS] token (borrowed from BERT) is prepended, and 1D positional embeddings are added to encode spatial ordering. This entire sequence is then processed by a standard Transformer encoder: alternating layers of multi-head self-attention and MLP blocks, with LayerNorm and residual connections. The [CLS] token’s final representation is passed to an MLP head for classification.

The key architectural innovation is the absence of any inductive bias — no convolutions, no local receptive fields, no pooling hierarchies. The model must learn all spatial structure from data. This is a weakness at small scale but a strength when pre-training data is abundant.

Key Concepts

  • Patch tokenization: the image-to-sequence conversion — splitting into P×P patches and projecting linearly — is the minimal operation needed to make images compatible with the Transformer input format.
  • CLS token: a learnable token prepended to the patch sequence whose final hidden state represents the whole image, following the convention established by BERT.
  • Position embeddings: standard 1D learnable embeddings; 2D-aware variants were explored but showed little improvement, suggesting the model learns spatial structure through attention patterns.
  • ViT-B/16, ViT-L/16, ViT-H/14: three model scales named by (size/patch-size). ViT-H/14 has 632M parameters and achieves the best accuracy.
  • Scale dependence: ViT underperforms CNNs on ImageNet alone (~76% vs ~77% for ResNet-50) but surpasses them after pre-training on ImageNet-21k (~14M images) or JFT-300M (~300M images).
  • Hybrid ViT: replaces raw patch embeddings with ResNet feature maps, combining convolutional and attention-based processing.

Results

When pre-trained on JFT-300M, ViT-H/14 achieves 88.55% top-1 accuracy on ImageNet, surpassing the best ResNet (BiT, 87.54%) and EfficientNet variants while requiring 2–4× less pre-training compute. On CIFAR-10 and CIFAR-100, ViT-L/16 achieves 99.7% and 97.8% respectively. On smaller pre-training datasets (ImageNet-21k), ViT-L/16 reaches 87.76% on ImageNet — still competitive with top CNNs but at lower cost.

The compute-accuracy tradeoff is ViT’s strongest argument: it matches state-of-the-art accuracy with substantially lower TPU-core-days of training.

Significance

ViT triggered a paradigm shift in computer vision. Before ViT, the dominant architectures were all convolutional (ResNet, EfficientNet, RegNet). After ViT, the field rapidly split into two camps — transformer-based and hybrid models — with much subsequent work on making ViT more data-efficient (DeiT uses knowledge distillation to train ViT on ImageNet-only), multi-scale (Swin Transformer introduces hierarchical windowed attention), and faster (linear attention approximations). ViT also became the backbone for large vision-language models (CLIP, ALIGN, Flamingo, LLaVA) and for generative vision systems.

The paper’s broader significance is architectural unification: the same Transformer block that processes text can process images, audio spectrograms, and video frames — enabling a common backbone architecture across modalities.

Graph View

Backlinks