AI/ML Wiki

Human-level control through deep reinforcement learning

2 min read

Contributions

  • Introduced DQN (Deep Q-Network): first system to achieve human-level performance across a diverse set of tasks (49 Atari games) using the same algorithm, architecture, and hyperparameters
  • Demonstrated that deep convolutional neural networks can serve as Q-function approximators for high-dimensional pixel input — bridging deep learning and RL
  • Two key stabilization techniques enabling stable Q-learning with neural networks:
    1. Experience replay: stores transitions in a replay buffer, breaks temporal correlations in training data
    2. Target network: separate frozen network for computing TD targets, updated periodically — stabilizes the moving target problem

Methodology

DQN combines Q-learning with a deep CNN operating on stacks of 4 grayscale frames (84×84 pixels). The Q-network outputs one Q-value per action (up to 18). Training:

  1. Store transition (s, a, r, s’) in replay buffer D
  2. Sample random minibatch from D
  3. Compute TD target: y = r + γ max_{a’} Q(s’, a’; θ⁻) using frozen target network θ⁻
  4. Minimize (y − Q(s,a;θ))² by SGD on online network θ
  5. Periodically copy θ → θ⁻

The replay buffer breaks the deadly correlation between consecutive training samples. The target network breaks the feedback loop between the prediction and the target, which previously caused divergence.

Results

  • Surpassed human performance on 29 of 49 Atari 2600 games
  • Achieved superhuman scores on games like Breakout, Pong, Space Invaders from pixels only
  • Same hyperparameters used across all games — demonstrated algorithm generality
  • Showed that the Q-values qualitatively track in-game events (value rises before scoring, drops before dying)

Limitations

  • Sample inefficiency: requires ~200 million frames per game (~38 days of play at 60fps)
  • Overestimation bias in Q-values due to max operator — addressed by Double DQN (van Hasselt et al. 2016)
  • Struggles with sparse reward, long horizons, and partial observability
  • Deterministic greedy policy — cannot solve stochastic games requiring mixed strategies
  • Limited to discrete action spaces (extended to continuous by DDPG)

Graph View

Backlinks