Nested Learning: The Illusion of Deep Learning Architectures

Contributions

Nested Learning (NL) Paradigm: A unified framework representing ML models as nested multi-level optimization problems, where each layer can be viewed as a learner with its own “context flow” (data, optimizer, objective).
Expressive Optimizers: Extends classical optimizers (SGD, Adam) to expressive associative memory systems. Shows that Adam/SGD are special cases of associative memory that compress gradient information.
Self-Modifying Learning Module (SMLM): A module that modifies its own learning rule during forward pass, enabling in-context learning without explicit meta-training.
Continuum Memory System: A memory architecture for continual learning, introducing the “Hope” model that consolidates knowledge online without catastrophic forgetting.

Nested Optimization: Each layer l solves: θ_l* = argmin_{θ_l} L_l(θ_l; x_l, A_l, f_l) where x_l is context flow, A_l is the optimizer, and f_l is the objective — all of which can themselves be learned.
Transformers as Nested Learners: Shows Transformers are a special case — attention layers perform extreme-frequency updates (effectively ∞ updates per token), while MLP layers perform zero updates (frozen parametric memory). This unifies the “Transformer as meta-learner” interpretation with classical architectures.
Hope Model: Hierarchical online persistent experience — a continual learning model derived from the NL paradigm with memory consolidation at test time.
Theoretical Analysis: Provides expressiveness bounds showing NL subsumes linear attention, Mamba, Transformers, and classical optimizers.

NL framework recovers Transformers, linear attention (Mamba, RWKV), and SGD/Adam as special instantiations
Hope model achieves competitive performance on continual learning benchmarks without replay buffers
Expressive optimizers outperform Adam on several few-shot meta-learning tasks
Theoretical analysis shows NL is strictly more expressive than standard deep learning architectures

Focuses on online memory consolidation; offline or batch consolidation strategies not addressed
Nested optimization increases computational cost at training time
Practical implementation of fully general NL requires careful engineering (e.g., differentiable optimizer selection)
Hope model evaluated on limited continual learning benchmarks; broader generalization unclear