Titans: Learning to Memorize at Test Time
Core Idea
Transformers forget: their attention window is fixed, so anything outside the context window is gone. Titans fixes this by adding a neural long-term memory module that learns what to remember during inference — not at training time. Memory is updated by gradient descent at test time, using a surprise-based signal: unexpected inputs are stored, expected inputs are discarded.
The architecture has three components mirroring human memory:
- Attention → short-term in-context memory (precise but limited window)
- Neural Long-Term Memory → persistent memory updated per token at inference
- MLP → parametric memory (frozen knowledge from pretraining)
Three Variants
| Variant | Integration |
|---|---|
| MAC (Memory as Context) | Memory output appended to attention context |
| MAG (Memory as Gate) | Memory gates the attention output |
| MAL (Memory as Layer) | Memory module replaces or supplements attention |
Significance
- Handles contexts up to 2M tokens — well beyond standard Transformer context windows
- Bridges the gap between attention-based models (precise but bounded) and recurrent models (unbounded but lossy)
- Surprise-based memory update is biologically plausible (cf. hippocampal novelty detection)
- Directly connects to Test-Time Compute (TTC) research: the memory learns at inference, not just retrieves
Relation to Nested Learning
The same authors’ Nested Learning paper provides a theoretical framework that subsumes Titans: attention = infinite-frequency learner, MLP = zero-frequency learner, Titans memory = intermediate-frequency learner between the two extremes.
Related Papers
- vaswani-2017-attention-is-all-you-need — Transformer baseline that Titans extends
- behrouz-2025-nested-learning-the-illusion — Theoretical unification of Titans under Nested Learning paradigm
- Mamba / RWKV — linear recurrent models that Titans competes with on long-context benchmarks