When we have $$m$$ i.i.d. samples $$\left\{x^{(1)},...,x^{(m)}\right\}$$ from a Normal distribution with parameters $$\mu$$ and $$\sigma$$, such that: $$p(x^{(i)};\mu, \sigma) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{1}{2\sigma^2}\left(x^{(i)}-\mu\right)^2\right)$$, an unbiased estimator for $$\mu$$ is the sample mean: $$\hat{\mu}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$$.

Johns Hopkins University

Google

A normal distribution, also known as a Gaussian distribution, is a continuous probability distribution defined by its mean $$\mu$$ and variance $$\sigma^2$$ (where $$\sigma$$ is the standard deviation). The probability density function is given by the formula: $$p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x - \mu)^2ight)$$. Changing the mean $$\mu$$ corresponds to a shift of the distribution along the $$x$$-axis, while increasing the variance $$\sigma^2$$ spreads the distribution out and lowers its peak.

Gaussian (Normal) Distribution

When we take $$n$$ i.i.d. samples from a Bernoulli distribution with parameter $$\theta$$, the mean of those samples is an unbiased estimator for $$\theta$$. Similarly, the mean of $$n$$ i.i.d. samples from a Normal distribution is an unbiased estimator for the $$\mu$$ parameter of the Normal distribution.

Examples of Unbiased Estimators

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

The mathematical formula for the normal distribution's probability density function can be directly implemented in Python. Using standard libraries like `math` and `numpy` (or their deep learning equivalents in PyTorch, JAX, or TensorFlow), a function can be defined to calculate the probability density $$p$$ for a given value $$x$$, given a mean $$\mu$$ and standard deviation $$\sigma$$. The code implementation is as follows:

```python
def normal(x, mu, sigma):
    p = 1 / math.sqrt(2 * math.pi * sigma**2)
    return p * np.exp(-0.5 * (x - mu)**2 / sigma**2)
```

This runnable implementation aligns with the formal definition: $$p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x - \mu)^2\right)$$.

Normal Distribution Probability Density Function Code

The Central Limit Theorem is a foundational mathematical principle in probability theory stating that the average of $$n$$ independent random samples drawn from any distribution (with mean $$\mu$$ and standard deviation $$\sigma$$) will approximately follow a normal distribution centered at the true mean $$\mu$$ with a standard deviation of $$\sigma/\sqrt{n}$$, as the sample size $$n$$ grows large. In machine learning, this theorem explains why the empirical error of a classifier evaluated on a test set converges to its true population error at an asymptotic rate of $$\mathcal{O}(1/\sqrt{n})$$.

Central Limit Theorem

The probability density function plot for a Gaussian distribution has a symmetric bell shape, where $$\sigma$$ defines the width of the curve and $$\mu$$ describes the center of the curve. The y-axis represents the probability density. The curve shown below illustrates a standard normal distribution, which has a mean of $$\mu = 0$$ and a standard deviation of $$\sigma = 1$$.

Plot of Normal Distribution Density Function

Unbiased Estimator for Normal $$\mu$$ Parameter

Gaussian noise is statistical noise that has a probability density function equal to that of the normal (Gaussian) distribution.

For example, when applying Gaussian noise to an image, the mean value of the noise determines the average change in pixel values. If the noise has a mean of -2, on average 2 is subtracted from each pixel. If the noise has a mean of 3, on average 3 is added to each pixel.

Gaussian Noise

An unbiased estimator for the variance $$\sigma^2$$ of a Normal distribution can be demonstrated through the expected value of the sample variance: $$\mathbb{E}[\hat{\sigma}^2_m] =\mathbb{E}\left[\frac{1}{m - 1}\sum_{i=1}^m \left(x^{(i)} - \hat{\mu}_m\right)^2\right]$$. See this [YouTube video](https://www.youtube.com/watch?v=D1hgiAla3KI) for a detailed explanation of this derivation.

Unbiased Estimator for Normal $$\sigma^2$$ Parameter

When we have $$m$$ i.i.d. samples $$\left\{x^{(1)}, \dots, x^{(m)}\right\}$$ from a Bernoulli distribution with parameter $$\theta$$, such that $$P(x^{(i)},\theta) = \theta^{x^{(i)}} (1-\theta)^{(1-x^{(i)})}$$, an unbiased estimator for $$\theta$$ is the mean of those samples:

$$\hat{\theta}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$$.

Learn Before

Related