The Bitter Lesson
Authors: Richard S. Sutton
Year: 2019
Source: Blog post (incompleteideas.net), March 13, 2019
Contributions
- Argued that the dominant lesson from 70 years of AI research is that general methods leveraging computation consistently outperform methods that encode human domain knowledge
- Surveyed major AI milestones (chess, Go, speech recognition, NLP) showing that in each case, scale and general learning beat hand-crafted expertise
- Articulated the principle: “The only thing that matters in the long run is search and learning”
- Warned AI researchers against the recurring temptation to hard-code domain structure rather than investing in scalable methods
Methodology
- Historical case study analysis across major AI subfields over seven decades
- Comparative analysis of knowledge-engineering approaches versus general learning/search approaches
- Identification of recurrent pattern: knowledge-engineering leads initially but is eventually surpassed by compute-scaled general methods
- Prescriptive argument: researchers should accept this lesson and redirect effort toward scalable general methods
Results
- Documented that every major long-term AI success has relied on general methods (deep learning, Monte Carlo search, statistical language modeling) rather than domain-specific engineering
- Showed that short-term gains from domain knowledge are repeatedly reversed when compute scales up
- Predicted that scaling general methods (search and learning) will continue to dominate specialized approaches
- The essay became widely cited as a conceptual foundation for the scaling hypothesis underlying large language models
Limitations
- Ignores sample efficiency: general methods often require orders of magnitude more data than human learners
- Does not address energy and compute costs of scaling, which raise economic and environmental concerns
- Critics argue data quality matters as much as scale, and that emergent capabilities from scaling are not fully understood
- The prescriptive conclusion (invest only in general methods) risks dismissing useful inductive biases entirely