When we have $m$ $i.i.d$ samples  $\{x^{(1)},...,x^{(m)}\}$ from a Bernoulli distribution with parameter $\theta$, such that: 
	$$P(x^{(i)},\theta) = \theta^{x^{(i)}} (1-\theta)^{(1-x^{i})}$$,
a unbiased estimator for $\theta$ is the mean of those samples:
	$\hat{\theta}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$


Unbiased Estimator for Bernoulli Parameter

See this [youtube video](https://www.youtube.com/watch?v=D1hgiAla3KI) for a explanation for why 
$$\mathbb{E}[\hat{\sigma}^2_m] =\mathbb{E}\left[\frac{1}{m - 1}\sum_{i=1}^m \left(x^{(i)} - \hat{\mu}_m\right)^2\right]$$
is an unbiased estimator for the variance $\sigma^2$. 

Unbiased Estimator for Normal $\sigma^2$ parameter

When we have $m$ $i.i.d$ samples  $\{x^{(1)},...,x^{(m)}\}$ from a Normal distribution with parameter $\mu$ and $\sigma$, such that: 
	$$P(x^{(i)};\mu, \sigma) = \frac{1}{\sigma\sqrt{2\pi}}e^{\frac{1}{2}(\frac{x-\mu}{\sigma})^2}$$,
a unbiased estimator for $\mu$ is the mean of those samples:
	$\hat{\mu}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$


Unbiased Estimator for Normal $\mu$ Parameter

When we take $n$ $i.i.d.$ samples from a Bernoulli distribution with parameter $\theta$, the mean of those samples is an unbiased estimator for $\theta$

Similarly, the mean of $n$ $i.i.d.$ samples from a Normal distribution is an unbiased estimator for the $\mu$ paramter of the Normal distirbution. 

Johns Hopkins University

An estimator $\hat{\theta}_m$ is called unbiased if $$\textrm{bias}(\hat{\theta}_m) = 0$$. 

By the definition of bias, $\mathrm{bias}(\hat \theta_m) = \mathbb{E}(\hat \theta_m) - \theta$, this implies $\mathbb{E}(\hat{\theta}_m) = \theta$. In other words, when a estimator is unbiased, the expected value of the estimator $\hat{\theta}_m$ is equal to the actual value of the parameter used to generate the data, $\theta$. 


Unbiased Estimators

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Examples of Unbiased Estimators

An estimator $\hat{\theta}_m$ is called biased if $$\textrm{bias}(\hat{\theta}_m) \not = 0$$. 

By the definition of bias, this implies $\mathbb{E}(\hat{\theta}_m) \not = \theta$. In other words, the expected value of the estimator $\hat{\theta}_m$ is not equal to the actual value of the paramter used to generate the data, $\theta$. 

Learn Before

Related

Learn After