Maximal entropy is on the left and max variance on the right (uses the assumption of a maximal and minimal value).

Maximal entropy v. Maximal variance (given max and min val)

University of Michigan - Ann Arbor

A measure of uncertainty based on the number of flips of a fair coin. Low entropy means little information is being revealed (i.e., sun rising in east) and high entropy means the outcomes are very uncertain but when actualized reveal a lot of information (i.e., a lottery ticket and reveal). For example, we would need to ask N questions to learn the sexes of N children. N questions would distinguish $2^N$ potential outcome sequences (N binary random events: N flips of a fair coin). Each outcome sequence has a probability of $$1\frac{1}{2^N}$$, which is converted to N = - $\log_2 (\frac{1}{2^N}) ^2$. If an outcome arises with probability p, it will have an uncertainty of $\log_2 (p) $ (an approximation of the # of yes/no questions needed to identify the outcome sequence).

Information Entropy

A distribution with constant probability.

Uniform Distribution

When the mean of the distribution is known, and it is also known that the values are all positive, we can assume exponential distribution. 

Exponential Distribution

Bell-shaped distribution, often referred to as Gaussian. To test the assumption of normality, a variety of methods are available, but the simplest is to inspect the data visually using a histogram or a Q-Q scatterplot. Real-world data are almost never perfectly normal, so this assumption can be considered reasonably met if the shape looks approximately symmetric and bell-shaped. The data in the example figure below is approximately normally distributed.

Normal Distribution

The Model Thinker

Claude Shann used four conditions to find a valid measure of entropy. Any measure of entropy would need to meet all four of these conditions.

Axiomatic foundations of entropy

Entropy can help distinguish Wolfram's four classes (i.e., equilibrium, cyclic (periodic), random, and complex). To do so, the information entropy must be calculate for different sub-sequences of the time-series data.

Distinguishing Wolfram's 4 classes of outcomes using entropy

Maximal Entropy Distributions

https://mathworld.wolfram.com/UniformDistribution.html

Uniform Distribution on Wolfram Mathworld

A uniform distribution can be assumed if nothing is known other than the range / set of probabilities.

Principle of Indifference

Paper Describing Central Limit Theorem and Its Connection to Normality: Kwak, S. G., & Kim, J. H. (2017). Central limit theorem: The cornerstone of modern statistics. Korean Journal of Anesthesiology, 70(2), 144. doi:10.4097/kjae.2017.70.2.144. Link: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5370305/

Reference Paper for Central Limit Theorem 

In probability theory, a log-normal (or lognormal) distribution is a continuous probability distribution of a random variable whose logarithm is normally distributed. Thus, if the random variable X is log-normally distributed, then Y = ln(X) has a normal distribution. Equivalently, if Y has a normal distribution, then the exponential function of Y, X = exp(Y), has a log-normal distribution. A random variable which is log-normally distributed takes only positive real values. It is a convenient and useful model for measurements in exact and engineering sciences, as well as medicine, economics and other topics (e.g., energies, concentrations, lengths, financial returns and other metrics).


Log-normal distribution

When we have $m$ $i.i.d$ samples  $\{x^{(1)},...,x^{(m)}\}$ from a Normal distribution with parameter $\mu$ and $\sigma$, such that: 
	$$P(x^{(i)};\mu, \sigma) = \frac{1}{\sigma\sqrt{2\pi}}e^{\frac{1}{2}(\frac{x-\mu}{\sigma})^2}$$,
a unbiased estimator for $\mu$ is the mean of those samples:
	$\hat{\mu}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$


Unbiased Estimator for Normal $\mu$ Parameter

The Central Limit Theorem is a foundational mathematical principle in probability theory stating that the average of $$n$$ independent random samples drawn from any distribution (with mean $$\mu$$ and standard deviation $$\sigma$$) will approximately follow a normal distribution centered at the true mean $$\mu$$ with a standard deviation of $$\sigma/\sqrt{n}$$, as the sample size $$n$$ grows large. In machine learning, this theorem explains why the empirical error of a classifier evaluated on a test set converges to its true population error at an asymptotic rate of $$\mathcal{O}(1/\sqrt{n})$$.

Learn Before

Related

Learn After