A Mathematical Theory of Communication
Authors: Claude E. Shannon
Year: 1948 | Bell System Technical Journal
Source: shannon-1948-a-mathematical-theory-of-communication
What it is
Shannon’s 1948 paper created information theory — the mathematical framework for quantifying information, measuring channel capacity, and proving the fundamental limits of data compression and reliable communication. It is arguably the most important engineering paper of the 20th century.
The key problem Shannon solved
Before Shannon, communication engineers designed telegraph and telephone systems by intuition and experience. There was no way to answer basic questions:
- How much information can a noisy telephone line carry?
- What is the minimum number of bits needed to represent a message?
- Is reliable communication over a noisy channel even possible?
Shannon answered all three — with mathematical proofs.
Information entropy
Shannon defined the entropy of a discrete source as:
H = −Σᵢ pᵢ log₂ pᵢ (bits)
where pᵢ is the probability of symbol i.
Intuition: H measures average surprise. A certain event (p=1) contributes 0 bits. A rare event (p very small) contributes many bits. The logarithm makes independent sources additive.
Examples:
- Fair coin flip: H = 1 bit
- English letters (uniform): H = log₂(26) ≈ 4.7 bits
- English letters (actual frequencies): H ≈ 4.14 bits
- English text with context: H ≈ 1.0–1.5 bits/character
The gap between 4.7 and 1.0 bits is redundancy — the structure of English allows compression.
Source coding theorem (lossless compression)
You cannot compress a source below its entropy H without losing information. The best possible lossless compression achieves an average code length approaching H bits per symbol.
This is why ZIP compression works — it exploits the statistical structure (redundancy) of files. But it cannot compress random data (H = 8 bits/byte for truly random bytes).
Connection to AI: the cross-entropy loss used to train language models is:
L = −Σᵢ p_data(xᵢ) log p_model(xᵢ)
This is exactly Shannon entropy — minimizing cross-entropy loss = making the model’s probability distribution approach the data distribution.
Channel capacity
A noisy channel has a maximum rate C at which information can be transmitted reliably:
C = W log₂(1 + S/N) bits per second
(Shannon-Hartley theorem: W = bandwidth, S/N = signal-to-noise ratio)
Shannon’s noisy channel coding theorem (the most surprising result):
- For any rate R < C: there exist codes that allow transmission with arbitrarily small error probability
- For any rate R > C: reliable communication is impossible
Before Shannon, engineers believed that more noise always meant more errors. Shannon proved that you can achieve zero error (in the limit) at any rate below capacity, just by using clever codes. Error-correcting codes (used in Wi-Fi, 5G, DVDs, space probes) are the practical realization.
The communication system model
Shannon’s abstract model:
Information Source → Transmitter → Channel ← Noise
↓
Receiver → Destination
This abstraction applies equally to:
- Telephone calls
- DNA replication
- Neural signaling in the brain
- Large language model training (data → model compression → inference)
Why it matters for AI
| Shannon’s concept | AI application |
|---|---|
| Entropy H | Cross-entropy loss, perplexity |
| Source coding | Model compression, quantization |
| Channel capacity | Information bottleneck in neural networks |
| Mutual information I(X;Y) | Feature selection, representation learning |
| Noisy channel coding | Noise-robust generation in LLMs |
Every time a language model reports perplexity, it’s computing an exponential of Shannon entropy. Every time you train with cross-entropy loss, you’re applying Shannon’s source coding theorem.
See also
- boole-1854-an-investigation-of-the-laws — Boolean algebra that Shannon also connected to circuits (1938)
- shannon-1938-symbolic-analysis-of-relay-and — Shannon’s earlier work on switching circuits
- turing-1936-on-computable-numbers-with-an — Computability (complementary theoretical foundation)