By the central limit theorem, as the size $$n$$ of a test dataset grows toward infinity, the empirical test error $$\epsilon_\mathcal{D}(f)$$ approaches the true population error $$\epsilon(f)$$ at an asymptotic convergence rate of $$\mathcal{O}(1/\sqrt{n})$$. This $$\mathcal{O}(1/\sqrt{n})$$ rate reveals that improving the precision of the error estimate is computationally expensive; for instance, estimating the test error twice as precisely requires collecting four times as many samples.

Asymptotic Convergence Rate of Test Error

The Central Limit Theorem is a foundational mathematical principle in probability theory stating that the average of $$n$$ independent random samples drawn from any distribution (with mean $$\mu$$ and standard deviation $$\sigma$$) will approximately follow a normal distribution centered at the true mean $$\mu$$ with a standard deviation of $$\sigma/\sqrt{n}$$, as the sample size $$n$$ grows large. In machine learning, this theorem explains why the empirical error of a classifier evaluated on a test set converges to its true population error at an asymptotic rate of $$\mathcal{O}(1/\sqrt{n})$$.

University of Michigan - Ann Arbor

Claude

Bell-shaped distribution, often referred to as Gaussian. To test the assumption of normality, a variety of methods are available, but the simplest is to inspect the data visually using a histogram or a Q-Q scatterplot. Real-world data are almost never perfectly normal, so this assumption can be considered reasonably met if the shape looks approximately symmetric and bell-shaped. The data in the example figure below is approximately normally distributed.

Normal Distribution

The Law of Large Numbers is a mathematical theorem guaranteeing that for repeated independent events, as the number of repetitions grows, the empirical average of the outcomes will converge to the true underlying expected probability. This principle mathematically ensures that consistent estimators become perfectly accurate given infinite data.

Law of Large Numbers

A normal distribution, also known as a Gaussian distribution, is a continuous probability distribution defined by its mean $$\mu$$ and variance $$\sigma^2$$ (where $$\sigma$$ is the standard deviation). The probability density function is given by the formula: $$p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x - \mu)^2ight)$$. Changing the mean $$\mu$$ corresponds to a shift of the distribution along the $$x$$-axis, while increasing the variance $$\sigma^2$$ spreads the distribution out and lowers its peak.

Gaussian (Normal) Distribution

Paper Describing Central Limit Theorem and Its Connection to Normality: Kwak, S. G., & Kim, J. H. (2017). Central limit theorem: The cornerstone of modern statistics. Korean Journal of Anesthesiology, 70(2), 144. doi:10.4097/kjae.2017.70.2.144. Link: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5370305/

Reference Paper for Central Limit Theorem 

Dive into Deep Learning

Maximal Entropy Distributions

In probability theory, a log-normal (or lognormal) distribution is a continuous probability distribution of a random variable whose logarithm is normally distributed. Thus, if the random variable X is log-normally distributed, then Y = ln(X) has a normal distribution. Equivalently, if Y has a normal distribution, then the exponential function of Y, X = exp(Y), has a log-normal distribution. A random variable which is log-normally distributed takes only positive real values. It is a convenient and useful model for measurements in exact and engineering sciences, as well as medicine, economics and other topics (e.g., energies, concentrations, lengths, financial returns and other metrics).


Log-normal distribution

When we have $m$ $i.i.d$ samples  $\{x^{(1)},...,x^{(m)}\}$ from a Normal distribution with parameter $\mu$ and $\sigma$, such that: 
	$$P(x^{(i)};\mu, \sigma) = \frac{1}{\sigma\sqrt{2\pi}}e^{\frac{1}{2}(\frac{x-\mu}{\sigma})^2}$$,
a unbiased estimator for $\mu$ is the mean of those samples:
	$\hat{\mu}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$


Unbiased Estimator for Normal $\mu$ Parameter

Central Limit Theorem

Probability density function plot for a gaussian distribution has got a symmetric bell shape where $\sigma$ defines the width of the curve and $\mu$ describes the center of the curve. The y axis is the probability density function.

The below curve is for a standard normal distribution having $\mu$ = 0  and $\sigma$ = 1


Plot of Normal Distribution Density Function

The mathematical formula for the normal distribution's probability density function can be directly implemented in Python. Using standard libraries like `math` and `numpy` (or their deep learning equivalents in PyTorch, JAX, or TensorFlow), a function can be defined to calculate the probability density $$p$$ for a given value $$x$$, given a mean $$\mu$$ and standard deviation $$\sigma$$. The code implementation is as follows:

```python
def normal(x, mu, sigma):
    p = 1 / math.sqrt(2 * math.pi * sigma**2)
    return p * np.exp(-0.5 * (x - mu)**2 / sigma**2)
```

This runnable implementation aligns with the formal definition: $$p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x - \mu)^2\right)$$.

Learn Before

Related

Learn After