AI/ML Wiki

Welcome to the Era of Experience

3 min read

Welcome to the Era of Experience

Authors: David Silver, Richard S. Sutton | Year: 2025

The next era of AI requires agents that learn from experience rather than imitating human data — just as AlphaZero surpassed human chess by playing against itself, future AI must generate its own training signal through interaction with rich environments.

Core Ideas

Silver and Sutton argue that AI has passed through two eras and is entering a third. The first era was hand-crafted knowledge: human experts encoded domain knowledge into rules, heuristics, and symbolic representations (GOFAI). The second era is the current one: learning from human data. Language models, image classifiers, and recommendation systems all learn by imitating patterns in data generated by humans. This era has produced remarkable systems, but Silver and Sutton argue it has a fundamental ceiling: a system trained to imitate humans cannot exceed the performance of the best humans in the training data.

The third era — the “era of experience” — is learning from interaction with the world rather than from human demonstrations. The paradigm case is AlphaZero: it received only the rules of chess and Go, then played millions of games against itself, generating its own training signal through experience. It surpassed every human player and every human-data-trained system because it was not bounded by what humans know how to do. Silver and Sutton argue that this paradigm — agents that explore, act, receive feedback, and improve — is the path to AI that genuinely exceeds human capability in open-ended domains.

The key requirements for experience-based agents are: (1) Rich environments — simulated or real worlds with enough complexity to generate useful training signal; (2) Scalable reward functions — automatic feedback signals that do not require human labeling of each interaction; (3) Long-horizon credit assignment — the ability to learn that an action taken now has consequences far in the future; (4) Exploration — the capacity to discover behaviors that are not in any training dataset.

The essay acknowledges that human data remains valuable for grounding concepts and providing a warm start, but argues it should be a scaffold rather than the primary training signal. The analogy is a human child who initially learns by imitating adults but then develops through their own exploration and experimentation.

Significance

The essay stakes out a research agenda for post-LLM AI development and represents the culmination of Sutton’s “bitter lesson” argument: if general methods leveraging computation always win, then experience-based RL — which generates unlimited training signal from self-play and environmental interaction — is the logical end point. It frames the limitations of current LLMs as a fundamental consequence of the imitation paradigm, not just an engineering gap.

The essay connects Bellman’s dynamic programming framework (the theoretical foundation of RL) to the most advanced systems (AlphaZero, Gemini, Claude) and argues that the next major capability jump requires going back to RL’s roots rather than scaling imitation learning further.

  • Silver et al. 2016 — AlphaGo, the proof-of-concept that experience-based RL can reach superhuman performance
  • Bellman 1957 — the mathematical foundation of RL that the era-of-experience agenda builds on
  • Sutton 2019 — the historical argument that general methods beat domain knowledge, of which this essay is the logical continuation
  • Sutton 1988 — TD learning, the core RL algorithm that experience-based agents will use at scale

Graph View

Backlinks