AI/ML Wiki

Prediction and Entropy of Printed English

4 min read

Prediction and Entropy of Printed English

Authors: Claude E. Shannon | Year: 1951

Measured the entropy of English text experimentally using human next-character prediction, finding ~1 bit/character and ~75% redundancy — directly foreshadowing the next-token prediction objective of modern language models.

Core Ideas

Shannon approached language as an information source and asked: how much information does each character in written English actually carry? To measure this, he devised an experimental method. Human subjects were shown a sequence of text and asked to guess the next character; the number of guesses required before a correct answer revealed how predictable (or surprising) each character was. This is equivalent to empirically measuring the conditional entropy H(X_n | X_{1}, …, X_{n-1}) — the uncertainty about the next symbol given all prior context.

The results showed that English is highly predictable. Given enough context, a human guesser needed very few attempts per character on average. Shannon estimated the entropy at roughly 1 bit per character, implying that English carries approximately 75% redundancy — three-quarters of each character is, in principle, predictable from what came before. This redundancy is why we can read garbled text, why compression algorithms work, and why spellcheckers are possible.

Shannon also showed that entropy decreases as context length grows: with more prior text, the next character becomes more predictable. This reveals that English has long-range statistical dependencies — structure that extends over many characters, not just adjacent pairs. The entropy rate only asymptotically approaches its true minimum as context length increases without bound.

A complementary compression-based approach cross-validated the human-subject results by bounding the entropy between approximately 0.6 and 1.3 bits/character, consistent with the prediction experiments. Together, these methods gave both upper and lower bounds on the true entropy of English text.

Significance

This paper is the conceptual origin of language modeling as a scientific and engineering task. Shannon framed the core question: given a sequence of characters, how well can the next one be predicted? This is identical — at the character level — to the next-token prediction objective used to train GPT, LLaMA, and every autoregressive language model today.

The connection to modern LLMs is direct and non-trivial. When GPT-3 (Brown et al. 2020) is trained to minimize cross-entropy loss on next-token prediction, it is operationalizing Shannon’s measurement procedure at scale: the model learns to compress language, and compression quality measures how well statistical structure has been captured. The loss value at convergence is an empirical estimate of the entropy rate of the training corpus — exactly what Shannon was measuring with human subjects in 1951.

The scaling laws (Kaplan et al. 2020) that govern LLM performance can be understood as measuring how close large models come to the Shannon entropy limit of natural language: more parameters and more data push the model’s per-token cross-entropy toward the theoretical minimum. Shannon’s result that entropy decreases with context length also explains why the Transformer architecture (Vaswani et al. 2017) benefits from long context windows — more context means lower entropy, tighter predictions, better compression.

The 75% redundancy figure has implications for model efficiency: language models are, in a sense, learning to exploit this redundancy. Techniques like speculative decoding (where a small model predicts tokens that a large model verifies) are practical consequences of this high redundancy. Shannon’s 1951 paper is not merely a historical curiosity — it is the theoretical floor that defines what language models are trying to achieve.

  • Shannon 1948 — entropy and information theory framework that this paper applies to natural language
  • Brown et al. 2020 (GPT-3) — large-scale next-token prediction; operationalizes Shannon’s prediction experiment at massive scale
  • Vaswani et al. 2017 — Transformer architecture enabling long-context next-token prediction
  • Kaplan et al. 2020 — scaling laws measure how close LLMs approach the Shannon entropy limit
  • Wiener 1948 — contemporaneous work on prediction and filtering in stochastic processes

Graph View

Backlinks