Contributions
- Introduced the REINFORCE algorithm: the first principled policy gradient method for connectionist (neural network) systems
- Derived the score function estimator (log-derivative trick) showing that ∇E[R] is computable without knowing the reward function’s derivative
- Established that direct policy parameterization — learning π(a|s;θ) without an action-value function — is a viable training objective
- Foundation for all modern policy optimization methods: Actor-Critic, PPO, TRPO, SAC
Methodology
REINFORCE updates policy parameters θ by gradient ascent on expected return:
∇J(θ) = E[∇log π(a_t | s_t; θ) · R_t]
The log-derivative trick allows computing this gradient without differentiating through the reward function itself — only through the policy. In practice, an unbiased gradient estimate is formed by rolling out episodes and weighting the log-probabilities of taken actions by their cumulative return.
Variance reduction techniques introduced in the paper:
- Baseline subtraction: replace R with (R − b) where b is any function of state — expectation unchanged, variance reduced
- Compatible function approximation: conditions under which a critic doesn’t bias the gradient estimate
Results
- Proved unbiasedness of the REINFORCE gradient estimator
- Demonstrated successful training on small connectionist tasks including recurrent networks
- Showed that policy gradient methods can be applied to non-differentiable reward signals (unlike supervised learning)
Limitations
- High variance: Monte Carlo return estimates are noisy; requires many episodes per update
- Sample inefficiency: on-policy — each batch of rollouts used once then discarded
- Baseline subtraction helps but doesn’t fully solve the variance problem; addressed later by Actor-Critic methods (which use a learned value function as baseline)
- Slow convergence on long-horizon tasks; superseded in practice by PPO, TRPO, and SAC