AI/ML Wiki

hochreiter-1997-long-short-term-memory

4 min read

Source Summary: Hochreiter & Schmidhuber 1997 — LSTM

Full title: Long Short-Term Memory
Authors: Sepp Hochreiter, Jürgen Schmidhuber
Affiliation: Technische Universität München (Hochreiter); IDSIA, Lugano (Schmidhuber)
Venue: Neural Computation, Vol. 9, No. 8, pp. 1735–1780 (1997)
PDF: papers/hochreiter-1997-long-short-term-memory.pdf

Contributions

  1. LSTM architecture — recurrent network with memory cells, input gates, output gates, and forget gates that maintain a “constant error carousel” to prevent vanishing/exploding gradients.
  2. Vanishing gradient analysis — formal proof that conventional backpropagation through time (BPTT) produces exponentially decaying error signals, making long-range dependencies unlearnable.
  3. Constant error flow — LSTM maintains error signals at exactly 1.0 through memory cells, enabling gradients to flow arbitrarily far back in time.
  4. Long time lags — LSTM successfully learned tasks requiring memory of inputs >1000 time steps back — far beyond what any prior RNN algorithm achieved.

The Vanishing Gradient Problem

Conventional BPTT

Error signal flowing backwards through time:

δ(t) = σ'(net(t)) · Σⱼ wⱼ · δ(t+1)

The total gradient after T time steps:

∂E/∂w(t) = ∏ᵢ₌ₜᵀ σ'(netᵢ) · wᵢ

If |σ’·w| < 1 throughout: the product → 0 exponentially → vanishing gradients
If |σ’·w| > 1 throughout: the product → ∞ exponentially → exploding gradients

For sigmoid activation σ, σ’ ≤ 0.25, so unless all weights are > 4, gradients vanish. This was first formally analyzed by Hochreiter in his 1991 diploma thesis.

The LSTM Architecture

Memory cell

The core unit: a cell state C(t) that flows through time with minimal modification:

C(t) = fₜ · C(t−1) + iₜ · g̃(t)

where:

  • C(t): cell state (the “memory”)
  • fₜ: forget gate — learned scalar in (0,1), controls how much of the past to remember
  • iₜ: input gate — learned scalar in (0,1), controls how much new information enters
  • g̃(t): candidate cell state (new content to potentially add)

Output

h(t) = oₜ · tanh(C(t))

where oₜ is the output gate — controls how much of the cell state is exposed.

Gate equations

fₜ = σ(Wf · [h(t−1), x(t)] + bf)   # forget gate
iₜ = σ(Wi · [h(t−1), x(t)] + bi)   # input gate
g̃(t) = tanh(Wg · [h(t−1), x(t)] + bg)  # candidate
oₜ = σ(Wo · [h(t−1), x(t)] + bo)   # output gate

Why this works

When fₜ = 1 and iₜ = 0, C(t) = C(t−1) — the cell state is unchanged.
The gradient ∂C(t)/∂C(t−1) = fₜ ≈ 1 → gradient flows back without decay.
The gates learn when to remember, forget, and expose information.

Experiments

All experiments used artificially constructed tasks designed to require long-range memory:

Experiment 1: Latch problem

Input sequence with embedded signal at t=1; output required at t=T. LSTM correctly remembered the signal over lags of 100–1000 steps. All prior methods failed.

Experiment 2: Embedded Reber grammar

Recognize strings from a grammar where the relevant decision depends on a symbol seen T steps earlier. LSTM solved lags up to 1000. BPTT, RTRL, Elman nets all failed.

Experiment 3: Adding problem

Two random sequences; sum two values at indicated positions separated by T steps. LSTM solved T up to 100; other methods failed at T > 10.

Experiment 4: Multiplication problem

Like adding but multiplicative — even harder for conventional RNNs.

Benchmark comparison

MethodMax lag solved
BPTT~10 steps
RTRL~10 steps
Elman~10 steps
Recurrent Cascade Correlation~100 steps (some tasks)
LSTM>1000 steps

Key Design Choices

No forget gate in original paper

The original 1997 LSTM did not include a forget gate — it was added by Gers et al. (2000). Without it, the cell state can only accumulate, never reset.

Gradient truncation

Error is not propagated through the input and output gates’ internal activations (only through the cell state path). This truncation does not affect long-term gradient flow but avoids computational cost.

O(W) per time step

Computation scales linearly with number of weights, same as a feedforward pass — efficient enough for practical use.

Key References

  • Hochreiter 1991 — diploma thesis: formal analysis of vanishing gradient problem
  • Williams & Zipser 1992 — BPTT algorithm
  • Elman 1990 — Elman (simple recurrent) networks
  • Graves & Schmidhuber 2005 — LSTM for sequence labeling (practical NLP application)
  • Vaswani et al. 2017 — Transformer: replaced LSTM for most NLP tasks (2017+)

Graph View

Backlinks