When we take $n$ $i.i.d.$ samples from a Bernoulli distribution with parameter $\theta$, the mean of those samples is an unbiased estimator for $\theta$

Similarly, the mean of $n$ $i.i.d.$ samples from a Normal distribution is an unbiased estimator for the $\mu$ paramter of the Normal distirbution. 

Examples of Unbiased Estimators

An estimator $\hat{\theta}_m$ is called biased if $$\textrm{bias}(\hat{\theta}_m) \not = 0$$. 

By the definition of bias, this implies $\mathbb{E}(\hat{\theta}_m) \not = \theta$. In other words, the expected value of the estimator $\hat{\theta}_m$ is not equal to the actual value of the paramter used to generate the data, $\theta$. 

Biased Estimators

An estimator $\hat{\theta}_m$ is called unbiased if $$\textrm{bias}(\hat{\theta}_m) = 0$$. 

By the definition of bias, $\mathrm{bias}(\hat \theta_m) = \mathbb{E}(\hat \theta_m) - \theta$, this implies $\mathbb{E}(\hat{\theta}_m) = \theta$. In other words, when a estimator is unbiased, the expected value of the estimator $\hat{\theta}_m$ is equal to the actual value of the parameter used to generate the data, $\theta$. 


Johns Hopkins University

$\mathrm{bias}(\hat \theta_m) = \mathbb{E}(\hat \theta_m) - \theta$, 

where $\hat{\theta}_m$ is the the estimator of a paramter, $\theta$ is the true value of that parameter, and $m$ is the sample size/number of data points. 

The bias of an estimator for a parameter is the difference between the expected value that the estimator will take when trained on data and the actual value of the parameter used to generate the data.

Mathematical Defintion of Bias

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Learn Before

Related

Learn After