Prediction and Entropy of Printed English

Authors: Claude E. Shannon
Year: 1951
Source: Bell System Technical Journal, 30(1), 50–64

---

Contributions

• Estimated the entropy of English text at approximately 1 bit per character (redundancy ~75%)
• Introduced a human-subject prediction method: subjects guess each next character in sequence; prediction difficulty measures entropy
• Showed that the statistical structure of language can be quantified as an information-theoretic measure
• Directly prefigures LLM next-token prediction: the task is conceptually identical to training language models on next-character/token prediction

Methodology

Experimental approach using human subjects. Participants were shown a sequence of text and asked to guess the next character one at a time; the number of guesses required before a correct answer was recorded. This procedure is equivalent to measuring the conditional entropy H(X_n | X_1, …, X_{n-1}) empirically. A complementary compression-based method was used to obtain both upper and lower bounds on the entropy rate, cross-validating the human-subject results.

Results

• Upper bound on English entropy: approximately 1.3 bits/character
• Lower bound: approximately 0.6–1.3 bits/character depending on context length used
• English is highly redundant: roughly 75% of characters are predictable from context
• Entropy decreases (predictability increases) as context window length grows, suggesting long-range statistical dependencies in natural language

Limitations

• Small human subject pool; results may not generalize across readers or text genres
• Human memory acts as a limited context window, likely underestimating long-range dependencies
• Only English was studied; other languages may have very different entropy rates
• Character-level analysis; does not capture semantic or syntactic structure above the character level