A fundamental theorem of information theory establishes a strict lower bound on data compression. It states that to encode data drawn randomly from a given probability distribution $$P$$, an observer needs at least $$H[P]$$ nats (or bits, if using base 2) to encode it, where $$H[P]$$ is the entropy of the distribution.

Claude

Entropy

Dive into Deep Learning

A measure of uncertainty based on the number of flips of a fair coin. Low entropy means little information is being revealed (i.e., sun rising in east) and high entropy means the outcomes are very uncertain but when actualized reveal a lot of information (i.e., a lottery ticket and reveal). For example, we would need to ask N questions to learn the sexes of N children. N questions would distinguish $2^N$ potential outcome sequences (N binary random events: N flips of a fair coin). Each outcome sequence has a probability of $$1\frac{1}{2^N}$$, which is converted to N = - $\log_2 (\frac{1}{2^N}) ^2$. If an outcome arises with probability p, it will have an uncertainty of $\log_2 (p) $ (an approximation of the # of yes/no questions needed to identify the outcome sequence).

Learn Before

Related