In information theory, the predictability of data is inversely related to the amount of information required to communicate it. A data stream that is highly predictable (e.g., repeating identical tokens) conveys little to no new information and is therefore easy to compress. Conversely, unpredictable events carry more information and are harder to compress.

Claude

Information theory is the mathematical understanding of parameters and rules of transmitting and processing information through communication systems.

Information Theory

Dive into Deep Learning

-    Books
-    Articles
-    Research papers

Information Theory references

A measure of uncertainty based on the number of flips of a fair coin. Low entropy means little information is being revealed (i.e., sun rising in east) and high entropy means the outcomes are very uncertain but when actualized reveal a lot of information (i.e., a lottery ticket and reveal). For example, we would need to ask N questions to learn the sexes of N children. N questions would distinguish $2^N$ potential outcome sequences (N binary random events: N flips of a fair coin). Each outcome sequence has a probability of $$1\frac{1}{2^N}$$, which is converted to N = - $\log_2 (\frac{1}{2^N}) ^2$. If an outcome arises with probability p, it will have an uncertainty of $\log_2 (p) $ (an approximation of the # of yes/no questions needed to identify the outcome sequence).

Information Entropy

A nat is a unit of information or entropy, analogous to a bit, but based on natural logarithms (base $$e$$) rather than base 2. The relationship between nats and bits is defined by the base change: one nat is equivalent to $$\frac{1}{\log(2)} \approx 1.44$$ bits.

Learn Before

Related