Long Short-Term Memory
Authors: Sepp Hochreiter, Jürgen Schmidhuber
Year: 1997 | Neural Computation
Source: hochreiter-1997-long-short-term-memory
What it is
LSTM solved the central failure mode of recurrent neural networks: the vanishing gradient problem. Conventional RNNs can’t learn dependencies spanning more than about 10–20 time steps because the error signal decays exponentially as it propagates backward. LSTM introduces memory cells with gated access that maintain gradients at constant strength, enabling learning over sequences of 1000+ steps.
The vanishing gradient problem
Backpropagation through time (BPTT) computes gradients across each time step. After T steps:
∂E/∂w(t=0) = ∏ᵢ₌₀ᵀ (∂hᵢ/∂hᵢ₋₁)
Each term ∂hᵢ/∂hᵢ₋₁ = σ’(netᵢ) · W. For sigmoid activation, σ’ ≤ 0.25. Unless all weights are > 4, the product → 0 exponentially. Error effectively dies before reaching long-past inputs.
Result: conventional RNNs (Elman, Jordan, BPTT) can only reliably use information from the last ~10 time steps.
The LSTM solution: constant error flow
The key idea: instead of letting gradients flow through a sequence of multiplications, route them through an additive path that doesn’t shrink them.
The memory cell C(t) updates as:
C(t) = f(t) · C(t−1) + i(t) · g̃(t)
The gradient ∂C(t)/∂C(t−1) = f(t). When the forget gate f(t) ≈ 1, the gradient flows back with no decay — a “constant error carousel.”
The gates
Three gates learned from data, each in (0,1):
f(t) = σ(Wf · [h(t−1), x(t)] + bf) # forget: how much to erase from cell
i(t) = σ(Wi · [h(t−1), x(t)] + bi) # input: how much new info to write
o(t) = σ(Wo · [h(t−1), x(t)] + bo) # output: how much cell to expose
New candidate:
g̃(t) = tanh(Wg · [h(t−1), x(t)] + bg)
Hidden state exposed to the rest of the network:
h(t) = o(t) · tanh(C(t))
Intuition: the memory cell is a conveyor belt
C(t−1) ─── × f(t) ──┬── C(t) ─── × f(t+1) ──→
│
+ i(t) · g̃(t)
The cell state carries information forward in time with minimal modification. The forget gate decides what to erase. The input gate decides what to write. The output gate decides what to read.
The gradient path ∂C(t)/∂C(t−k) = ∏ f(t−j) flows back through only the forget gates — not through tanh derivatives, not through full weight matrices. When f ≈ 1, this product stays ≈ 1 for arbitrary k.
Experimental results
LSTM solved artificial tasks requiring memory of 1000+ steps that every prior RNN algorithm had failed on:
| Task | Required memory span | LSTM | BPTT | RTRL | Elman |
|---|---|---|---|---|---|
| Latch problem | 100–1000 steps | ✓ | ✗ | ✗ | ✗ |
| Embedded Reber grammar | 100–1000 steps | ✓ | ✗ | ✗ | ✗ |
| Adding problem | 100 steps | ✓ | ✗ | ✗ | ✗ |
Why LSTM dominated NLP (1997–2017)
From the late 1990s through 2017, LSTM was the dominant architecture for:
- Speech recognition: sequence-to-sequence with attention
- Machine translation: encoder-decoder LSTMs
- Language modeling: predict next word
- Text generation: story writing, dialogue
- Time series: financial data, sensor readings
The combination of long-range memory + trainability made LSTM the first practically successful sequence model for variable-length inputs.
When Transformers replaced LSTM
The Transformer (Vaswani et al., 2017) largely replaced LSTM for NLP after 2018:
- Transformers attend to all positions simultaneously (O(n²) attention vs. O(n) LSTM)
- Transformers parallelize better — LSTMs are sequential by nature
- Transformers scale better to very long contexts with modern hardware
For tasks with truly long sequences (>10k tokens), Transformers also struggle — suggesting the vanishing gradient problem in attention has its own forms. Research on state space models (Mamba, 2023) revisits LSTM-style recurrence at modern scale.
See also
- rumelhart-1986-learning-representations-by-back-propagating-errors — Backpropagation that LSTM fixes for sequences
- vaswani-2017-attention-is-all-you-need — Transformer: the architecture that succeeded LSTM