Paper Describing Central Limit Theorem and Its Connection to Normality: Kwak, S. G., & Kim, J. H. (2017). Central limit theorem: The cornerstone of modern statistics. Korean Journal of Anesthesiology, 70(2), 144. doi:10.4097/kjae.2017.70.2.144. Link: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5370305/

University of Michigan - Ann Arbor

Bell-shaped distribution, often referred to as Gaussian. To test the assumption of normality, a variety of methods are available, but the simplest is to inspect the data visually using a histogram or a Q-Q scatterplot. Real-world data are almost never perfectly normal, so this assumption can be considered reasonably met if the shape looks approximately symmetric and bell-shaped. The data in the example figure below is approximately normally distributed.

Normal Distribution

Reference Paper for Central Limit Theorem 

Maximal Entropy Distributions

In probability theory, a log-normal (or lognormal) distribution is a continuous probability distribution of a random variable whose logarithm is normally distributed. Thus, if the random variable X is log-normally distributed, then Y = ln(X) has a normal distribution. Equivalently, if Y has a normal distribution, then the exponential function of Y, X = exp(Y), has a log-normal distribution. A random variable which is log-normally distributed takes only positive real values. It is a convenient and useful model for measurements in exact and engineering sciences, as well as medicine, economics and other topics (e.g., energies, concentrations, lengths, financial returns and other metrics).


Log-normal distribution

When we have $m$ $i.i.d$ samples  $\{x^{(1)},...,x^{(m)}\}$ from a Normal distribution with parameter $\mu$ and $\sigma$, such that: 
	$$P(x^{(i)};\mu, \sigma) = \frac{1}{\sigma\sqrt{2\pi}}e^{\frac{1}{2}(\frac{x-\mu}{\sigma})^2}$$,
a unbiased estimator for $\mu$ is the mean of those samples:
	$\hat{\mu}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$


Unbiased Estimator for Normal $\mu$ Parameter

The Central Limit Theorem is a foundational mathematical principle in probability theory stating that the average of $$n$$ independent random samples drawn from any distribution (with mean $$\mu$$ and standard deviation $$\sigma$$) will approximately follow a normal distribution centered at the true mean $$\mu$$ with a standard deviation of $$\sigma/\sqrt{n}$$, as the sample size $$n$$ grows large. In machine learning, this theorem explains why the empirical error of a classifier evaluated on a test set converges to its true population error at an asymptotic rate of $$\mathcal{O}(1/\sqrt{n})$$.

Learn Before

Related