Nested Learning: The Illusion of Deep Learning Architectures
Core Idea
Every deep learning architecture — Transformer, Mamba, RWKV, ResNet — appears to be a novel design, but they are all special cases of a single unified framework: Nested Learning (NL).
The NL paradigm represents any ML model as a nested hierarchy of optimization problems. Each layer is a learner with its own:
- Context flow x_l: what data it sees
- Optimizer A_l: how it updates (SGD, Adam, or something learned)
- Objective f_l: what it minimizes
The insight: layers don’t just transform data — they learn from it, each at a different frequency.
Transformers Demystified
Under NL, the Transformer’s two main components reveal their nature:
| Component | Update Frequency | Interpretation |
|---|---|---|
| Attention | ∞ (per token) | Extreme online learner — updates for every new token |
| MLP | 0 (frozen) | Frozen parametric memory — never updates at inference |
This explains why attention is so powerful for in-context learning (it sees everything) and why MLPs act as fixed knowledge stores.
Key Innovations Derived from NL
- Expressive Optimizers: Extends SGD/Adam as associative memory systems — classical optimizers are memory compressors operating on gradients
- Self-Modifying Learning Module (SMLM): A module that modifies its own learning rule during the forward pass, enabling in-context learning without meta-training
- Hope Model: A continual learning architecture derived from NL for online knowledge consolidation without catastrophic forgetting
Significance
- Provides a theoretical vocabulary for comparing architectures that previously seemed incomparable
- Predicts new architectures by exploring the space of (frequency, optimizer, objective) combinations
- “The illusion” in the title: different architectures feel distinct but are reparametrizations of the same optimization structure
- Unifies meta-learning (MAML, few-shot) with standard deep learning under one mathematical framework
Related Papers
- vaswani-2017-attention-is-all-you-need — Transformer: the canonical special case of NL with ∞/0 frequency split
- behrouz-2024-titans-learning-to-memorize — Titans: a practical NL instantiation with intermediate-frequency memory
- kaushik-2025-the-universal-weight-subspace — Empirical companion: why NL special cases share weight subspaces