For a sample of size $m$ from a Normally distributed random variable with paramters $\mu$ and $\sigma$, the sample variance, which is the sum of the squared deviations from the mean over the number of samples, the estimator
	$$\mathbb{E}[\hat{\sigma'}^2_m] = \mathbb{E}\left[\frac{1}{m}\sum_{i=1}^m \left(x^{(i)} - \hat{\mu}_m\right)^2\right]$$
Is a biased estimator for the variance $\sigma^2$. 

To see why, consider that the sample variance, 
	$$\mathbb{E}[\hat{\sigma}^2_m] =\mathbb{E}\left[\frac{1}{m - 1}\sum_{i=1}^m \left(x^{(i)} - \hat{\mu}_m\right)^2\right]$$
is an unbiased estimator for the variance $\sigma^2$. Then 
$\textrm{bias}(\hat{\sigma}_m^2 = \mathbb{E}\left[\frac{1}{m - 1}\sum_{i=1}^m \left(x^{(i)} - \hat{\mu}_m\right)^2\right] - \sigma^2 = 0$, so $ \mathbb{E}(\hat{\sigma}_m^2) = \mathbb{E}\left[\frac{1}{m - 1}\sum_{i=1}^m \left(x^{(i)} - \hat{\mu}_m\right)^2\right] = \sigma^2 \not =  \mathbb{E}\left[\frac{1}{m}\sum_{i=1}^m \left(x^{(i)} - \hat{\mu}_m\right)^2\right] = \mathbb{E}(\hat{\sigma'}_m^2)$, since $m-1 \not = m$. 


Example of Biased Estimators

An estimator $\hat{\theta}_m$ is called biased if $$\textrm{bias}(\hat{\theta}_m) \not = 0$$. 

By the definition of bias, this implies $\mathbb{E}(\hat{\theta}_m) \not = \theta$. In other words, the expected value of the estimator $\hat{\theta}_m$ is not equal to the actual value of the paramter used to generate the data, $\theta$. 

Johns Hopkins University

An estimator $\hat{\theta}_m$ is called unbiased if $$\textrm{bias}(\hat{\theta}_m) = 0$$. 

By the definition of bias, $\mathrm{bias}(\hat \theta_m) = \mathbb{E}(\hat \theta_m) - \theta$, this implies $\mathbb{E}(\hat{\theta}_m) = \theta$. In other words, when a estimator is unbiased, the expected value of the estimator $\hat{\theta}_m$ is equal to the actual value of the parameter used to generate the data, $\theta$. 


Unbiased Estimators

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

When we take $n$ $i.i.d.$ samples from a Bernoulli distribution with parameter $\theta$, the mean of those samples is an unbiased estimator for $\theta$

Similarly, the mean of $n$ $i.i.d.$ samples from a Normal distribution is an unbiased estimator for the $\mu$ paramter of the Normal distirbution. 

Learn Before

Related

Learn After