When we have $m$ $i.i.d$ samples  $\{x^{(1)},...,x^{(m)}\}$ from a Bernoulli distribution with parameter $\theta$, such that: 
	$$P(x^{(i)},\theta) = \theta^{x^{(i)}} (1-\theta)^{(1-x^{i})}$$,
a unbiased estimator for $\theta$ is the mean of those samples:
	$\hat{\theta}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$


Unbiased Estimator for Bernoulli Parameter

When we have $m$ $i.i.d$ samples  $\{x^{(1)},...,x^{(m)}\}$ from a Normal distribution with parameter $\mu$ and $\sigma$, such that: 
	$$P(x^{(i)};\mu, \sigma) = \frac{1}{\sigma\sqrt{2\pi}}e^{\frac{1}{2}(\frac{x-\mu}{\sigma})^2}$$,
a unbiased estimator for $\mu$ is the mean of those samples:
	$\hat{\mu}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$


Unbiased Estimator for Normal $\mu$ Parameter

An unbiased estimator for the variance $$\sigma^2$$ of a Normal distribution can be demonstrated through the expected value of the sample variance: $$\mathbb{E}[\hat{\sigma}^2_m] =\mathbb{E}\left[\frac{1}{m - 1}\sum_{i=1}^m \left(x^{(i)} - \hat{\mu}_m\right)^2\right]$$. See this [YouTube video](https://www.youtube.com/watch?v=D1hgiAla3KI) for a detailed explanation of this derivation.

Unbiased Estimator for Normal $$\sigma^2$$ Parameter

When we take $$n$$ i.i.d. samples from a Bernoulli distribution with parameter $$\theta$$, the mean of those samples is an unbiased estimator for $$\theta$$. Similarly, the mean of $$n$$ i.i.d. samples from a Normal distribution is an unbiased estimator for the $$\mu$$ parameter of the Normal distribution.

Johns Hopkins University

Google

An estimator $$\hat{\theta}_m$$ is called unbiased if $$\textrm{bias}(\hat{\theta}_m) = 0$$. By the definition of bias, $$\mathrm{bias}(\hat \theta_m) = \mathbb{E}(\hat \theta_m) - \theta$$, this implies $$\mathbb{E}(\hat{\theta}_m) = \theta$$. In other words, when an estimator is unbiased, the expected value of the estimator $$\hat{\theta}_m$$ is equal to the actual value of the parameter used to generate the data, $$\theta$$.

Unbiased Estimator

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

An estimator $\hat{\theta}_m$ is called biased if $$\textrm{bias}(\hat{\theta}_m) \not = 0$$. 

By the definition of bias, this implies $\mathbb{E}(\hat{\theta}_m) \not = \theta$. In other words, the expected value of the estimator $\hat{\theta}_m$ is not equal to the actual value of the paramter used to generate the data, $\theta$. 

Learn Before

Related

Learn After