Source Summary: LeCun 2022 — A Path Towards Autonomous Machine Intelligence
Full title: A Path Towards Autonomous Machine Intelligence, Version 0.9.2
Authors: Yann LeCun
Affiliation: NYU Courant Institute / Meta FAIR
Venue: OpenReview preprint (June 27, 2022)
PDF: papers/lecun-2022-a-path-towards-autonomous-machine.pdf
Contributions
- Critique of autoregressive LLMs — argues that next-token prediction is insufficient for intelligence: it cannot model physical reality, reason causally, or plan efficiently.
- JEPA (Joint Embedding Predictive Architecture) — a world model architecture that predicts in abstract representation space rather than pixel/token space, trained with self-supervised learning.
- Hierarchical planning with world models — a cognitive architecture where a world model supports multi-level planning: an agent imagines consequences before acting.
- Intrinsic motivation — proposes replacing external reward signals with intrinsic objectives (curiosity, prediction error reduction) as the driver of behavior.
The Central Argument
From the paper:
“Human and non-human animals seem able to learn enormous amounts of background knowledge about how the world works through observation and through an incomprehensibly small amount of interactions in a task-independent, unsupervised way.”
LLMs fail to achieve this because:
- They model sequences of tokens, not states of the world
- Token-level prediction forces the model to predict irrelevant details (exact wording) rather than structure
- They cannot represent physical causality or temporal dynamics
- They require massive amounts of labeled data for each new task
JEPA: Joint Embedding Predictive Architecture
The key architectural proposal:
Standard predictive model:
x(t) → Encoder → ŝ(t+1) compared against s(t+1)
(predicts raw future observation — too difficult, too detailed)
JEPA:
x(t) → Encoder → s(t)
s(t) + action → Predictor → ŝ(t+1) compared against Encoder(x(t+1))
(predicts abstract representation of future — tractable)
The encoder learns a representation that captures the structure of the future state, not its exact form. This is crucial: you don’t need to predict every pixel to predict that “the ball will be on the right side of the screen.”
H-JEPA (Hierarchical JEPA) stacks multiple JEPA modules at different levels of abstraction, enabling planning at multiple time horizons simultaneously.
Proposed Cognitive Architecture
LeCun proposes a complete agent architecture:
| Module | Function |
|---|---|
| Perception | Encode sensory input into abstract representation |
| World model | Predict future states from current state + action (JEPA) |
| Cost module | Evaluate states against intrinsic and extrinsic objectives |
| Actor | Select actions via optimization over the world model |
| Short-term memory | Current state representation |
| Long-term memory | Episodic memory; retrieved context |
The key claim: this architecture does not require reinforcement learning or massive labeled datasets. The world model trains self-supervised; the actor optimizes against it.
Why Not Generative Models?
LeCun argues against training world models that predict in raw observation space (pixels, tokens):
- Irrelevant details: the exact appearance of an object at t+1 is not predictable and not needed
- Mode collapse / over-smoothing: generative models that must predict exact output tend to average over uncertainty, producing blurry or low-quality predictions
- Computational cost: predicting full-resolution future states is expensive and unnecessary
JEPA avoids this by predicting in representation space — only the abstract structure of the future, not its surface form.
Connection to Causality and Symbolism
LeCun’s world model argument is essentially a connectionist implementation of Pearl’s causal reasoning (Rung 2): the JEPA predictor models state(t+1) = f(state(t), action), which is precisely the interventional query P(S_{t+1} | do(A_t=a)).
And the hierarchical architecture mirrors the symbolic AI ambition: an agent with explicit representations, a world model, and goal-directed planning — not just a pattern matcher.
Key References (from paper)
- LeCun, Chopra et al. 2006 — Energy-based models
- Hafner et al. 2020 — DreamerV1 (learned world model for RL)
- Bardes, Ponce, LeCun 2022 — VICReg (self-supervised pretraining for JEPA)
- Pearl 2000 — Causality (the theoretical framework this implements)