Claude Shann used four conditions to find a valid measure of entropy. Any measure of entropy would need to meet all four of these conditions.

University of Michigan - Ann Arbor

A measure of uncertainty based on the number of flips of a fair coin. Low entropy means little information is being revealed (i.e., sun rising in east) and high entropy means the outcomes are very uncertain but when actualized reveal a lot of information (i.e., a lottery ticket and reveal). For example, we would need to ask N questions to learn the sexes of N children. N questions would distinguish $2^N$ potential outcome sequences (N binary random events: N flips of a fair coin). Each outcome sequence has a probability of $$1\frac{1}{2^N}$$, which is converted to N = - $\log_2 (\frac{1}{2^N}) ^2$. If an outcome arises with probability p, it will have an uncertainty of $\log_2 (p) $ (an approximation of the # of yes/no questions needed to identify the outcome sequence).

Information Entropy

The Model Thinker

Axiomatic foundations of entropy

Entropy can help distinguish Wolfram's four classes (i.e., equilibrium, cyclic (periodic), random, and complex). To do so, the information entropy must be calculate for different sub-sequences of the time-series data.

Learn Before

Related