A measure of uncertainty based on the number of flips of a fair coin. Low entropy means little information is being revealed (i.e., sun rising in east) and high entropy means the outcomes are very uncertain but when actualized reveal a lot of information (i.e., a lottery ticket and reveal). For example, we would need to ask N questions to learn the sexes of N children. N questions would distinguish $2^N$ potential outcome sequences (N binary random events: N flips of a fair coin). Each outcome sequence has a probability of $$1\frac{1}{2^N}$$, which is converted to N = - $\log_2 (\frac{1}{2^N}) ^2$. If an outcome arises with probability p, it will have an uncertainty of $\log_2 (p) $ (an approximation of the # of yes/no questions needed to identify the outcome sequence).

Information Entropy

A fundamental theorem of information theory establishes a strict lower bound on data compression. It states that to encode data drawn randomly from a given probability distribution $$P$$, an observer needs at least $$H[P]$$ nats (or bits, if using base 2) to encode it, where $$H[P]$$ is the entropy of the distribution.

Limit of Data Compression in Information Theory

University of Michigan - Ann Arbor

Focuses of Physics

The Model Thinker

A property that must be transferred to perform some work or heat something up.

Energy

A system composed of many distributed elements whose aggregate properties cannot be predicted by observing the behavior of an element. Furthermore, the observable manifestations or 'forms' of a complex system may only represent superficial aspects, with its true 'essence' or fundamental nature remaining hidden or less apparent beneath the surface.

Learn Before

Related

Learn After