Communication in the Presence of Noise
Authors: Claude E. Shannon | Year: 1949
Proved the Nyquist-Shannon sampling theorem and derived the AWGN channel capacity formula C = B log₂(1 + S/N), unifying continuous and discrete communication theory.
Core Ideas
The central result of this paper is the sampling theorem: any signal bandlimited to B Hz is completely determined by its values at uniformly spaced discrete samples taken at a rate of at least 2B samples per second (the Nyquist rate). Between samples, the signal can be reconstructed perfectly using sinc interpolation. This result is the mathematical bridge that makes digital representation of continuous signals lossless in principle — every audio file, digital image, and video stream rests on it.
Shannon extended this to noisy channels by applying the entropy framework from his 1948 paper to continuous-time signals. Treating additive white Gaussian noise (AWGN) as the canonical noise model, he derived the Shannon-Hartley capacity formula: C = B log₂(1 + S/N). This states that the maximum error-free transmission rate over a noisy channel is a function of bandwidth B and signal-to-noise ratio S/N. Neither bandwidth nor SNR alone determines capacity — the trade-off between them follows logarithmic scaling.
A key implication of the capacity formula is the bandwidth-SNR trade-off. Doubling bandwidth doubles capacity linearly; doubling SNR only increases capacity by one bit. This asymmetry explains why modern wireless systems (5G, Wi-Fi 6) prioritize spectrum efficiency alongside raw power. The log scaling also explains why error-correcting codes can trade bandwidth for reliability: Shannon proved such codes must exist, even if he did not construct them.
The paper unified what had previously been separate lines of work on continuous and discrete communication. By showing that a bandlimited continuous signal is equivalent to a discrete sample sequence, Shannon reduced the continuous channel problem to the discrete framework developed in 1948. This unification made information theory a single coherent discipline applicable to all physical communication media.
Significance
The Nyquist-Shannon sampling theorem is the foundational theorem of all digital signal processing. Every analog-to-digital converter, codec, and digital communication protocol relies on it. The capacity formula C = B log₂(1 + S/N) defined the absolute limits of communication long before practical codes approached them — it took until turbo codes (1993) and LDPC codes to come within a few percent of Shannon capacity.
In the context of machine learning and AI, the sampling theorem is relevant wherever continuous signals are discretized as model inputs (audio waveforms, sensor data, images). The capacity formula appears implicitly in information-bottleneck analyses of neural networks, where the mutual information between layers is bounded by channel-theoretic arguments.
Related Papers
- Shannon 1948 — foundational paper establishing entropy and discrete channel capacity; this paper extends that framework to continuous channels
- Wiener 1948 — contemporaneous work on control and communication; Wiener developed spectral analysis of noise independently