AI/ML Wiki

schultz-1993-responses-of-monkey-dopamine-neurons

3 min read

Responses of Monkey Dopamine Neurons to Reward and Conditioned Stimuli

Authors: Wolfram Schultz, Paul Apicella, Tomas Ljungberg
Year: 1993 | Journal of Neuroscience
Source: schultz-1993-responses-of-monkey-dopamine-neurons

What it is

Schultz et al. recorded from dopamine neurons in monkeys as they learned a spatial memory task, and observed a key pattern: dopamine neurons respond to unexpected rewards early in learning, but this response transfers to the reward-predicting cue as the animal learns the association. This is the empirical observation that would become the neural signature of reinforcement learning.

The experiment

Monkeys were trained on three progressively harder spatial tasks:

  1. Spatial choice: instruction and trigger lights appear simultaneously — touch the indicated lever
  2. Instructed spatial: instruction appears 1 sec before trigger — remember the target, then move when triggered
  3. Delayed response: instruction appears for 1 sec then disappears — hold the target location in working memory for 2.5–3.5 sec before the trigger

Two dopamine-neuron populations were compared:

  • 76 neurons recorded during learning of each new task
  • 163 neurons recorded after performance was established

The key finding: reward responses diminish with learning

Stage% of DA neurons with reward response
During learning25%
After established performance9%

When the task is new and rewards are unexpected, dopamine neurons fire at the moment of reward delivery. As the animal masters the task and rewards become predictable, these reward responses largely disappear.

Meanwhile, responses to the conditioned stimuli (CS) that predict reward persist:

  • Instruction cue: 56% of neurons (learning), 44% (established)
  • Trigger cue: 36% (learning), 40% (established)

The dopamine response migrates backward in time — from reward to the earlier cues that predict reward.

What dopamine neurons do NOT do

One of the most important negative findings: no dopamine neuron showed sustained activity during the delay period between the instruction and trigger (2.5–3.5 seconds).

This is significant because:

  • Neurons in the striatum and prefrontal cortex DO show sustained delay-period activity (encoding the spatial memory)
  • If dopamine encoded “expectation” or “working memory,” you’d expect sustained activity
  • Instead, dopamine fires only at discrete events — the phasic response to stimuli

Conclusion: dopamine neurons encode events and their significance, not continuous state representations.

What this looked like in the data

Early learning (reward unexpected):

Instruction   Delay   Trigger   Reward
    |           ...      |        |
    CS fire             CS fire   DA BURST ←
    
Late learning (reward predicted):

Instruction   Delay   Trigger   Reward
    |           ...      |        |
    DA BURST            DA BURST  (no burst)
    ↑ transferred

The dopamine burst moves from the reward to the predictive cue. When the animal fully expects the reward, the dopamine signal at reward time disappears.

Connection to reinforcement learning theory

Although the 1993 paper doesn’t use the TD learning vocabulary, the data pattern maps directly onto the temporal difference prediction error:

  • Dopamine burst at reward = positive prediction error (reward was better than expected)
  • Dopamine burst at CS = positive prediction error at the CS (CS predicts reward that wasn’t predicted before)
  • Loss of dopamine at reward = prediction error → 0 (reward now fully predicted)

This connection would be made explicit by Schultz, Dayan & Montague (1997).

Why the paper matters

This was the crucial empirical step connecting dopamine neuroscience to learning theory:

  1. Showed that dopamine signals are not simply “reward” — they are teaching signals that update with experience
  2. Demonstrated that the signal is phasic (event-driven), not tonic (state-encoding)
  3. Established that DA neurons respond to predictive cues, not just unconditioned rewards

These properties — transient, error-like, transferable to predictors — are exactly what a reinforcement learning system needs.

See also

Graph View

Backlinks