$$\textrm{bias}(\hat{\theta}_m) = \mathbb{E}[\hat{\theta}_m] - \theta \\
= \mathbb{E}\left[\frac{1}{m} \sum_{i=1}^{m}x^{(i)}\right] - \theta \\
	= \frac{1}{m} \sum_{i=1}^{m}\mathbb{E}\left[x^{(i)}\right] - \theta \\
	= \frac{1}{m} \sum_{i=1}^{m}\sum_{x^{(i)} = 0}^1\left( x^{(i)}\theta^{x^{(i)}}(1-\theta)^{(1-x^{(i)})}\right)- \theta \\
	= \frac{1}{m} \sum_{i=1}^m(\theta) - \theta \\
	= \theta - \theta = 0$$

Reasons that the steps above work:  The first one is the definition of bias. The second one is plugging in our estimator. The third one is because the expected value of the sum is the sum of the expected values. The fourth is by the definition of expected value of a discrete random variable (it is each value times the probability at that value, which for a bernoulli r.v. is $\theta$ if $x = 1$, and $1 - \theta$ if $x = 0$, which can be summarized as $P(x;\theta) = \theta^x(1-\theta)^{1-x}$. The fifth line works because we have assumed identically distributed random variables (note that we also assumed that they are independent, but this isn’t necessary: the same derivation holds when the random variables are dependent). Then getting from the fifth to the sixth lines requires recognizing that the sum of $m$ $\theta$'s is equal to $m*\theta$, and when you multiply that by $\frac{1}{m}$ you get $\theta$. 

Proof that the Mean of Bernoulli samples is an Unbiased Estimator for the Bernoulli Parameter $\theta$ 

When we have $m$ $i.i.d$ samples  $\{x^{(1)},...,x^{(m)}\}$ from a Bernoulli distribution with parameter $\theta$, such that: 
	$$P(x^{(i)},\theta) = \theta^{x^{(i)}} (1-\theta)^{(1-x^{i})}$$,
a unbiased estimator for $\theta$ is the mean of those samples:
	$\hat{\theta}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$


Johns Hopkins University

When we take $n$ $i.i.d.$ samples from a Bernoulli distribution with parameter $\theta$, the mean of those samples is an unbiased estimator for $\theta$

Similarly, the mean of $n$ $i.i.d.$ samples from a Normal distribution is an unbiased estimator for the $\mu$ paramter of the Normal distirbution. 

Examples of Unbiased Estimators

Bernoulli distribution is a distribution over a single binary variable. It can only have results in 0 or 1. The sum of the probability of 0  and the probability of 1 is equal to 1. If we denote the probability of 0 as $\sigma$, Bernoulli distribution then has a mean of $\sigma$ and a variance of  $\sigma * (1- \sigma)$.

Bernoulli Distribution

When we talk about a distribution, there are some things we might be interested in:
We assume this Bernoulli Distribution has a probabilty p, for n=1 ("success") and 1-p for n=0("failure")
N is the number of trails
1) Probability density function: $ P(N)=p^{N} *(1-p)^{1-N} $
2) Expectation: $ E(X)=p*1+(1-p)*0=p $
3) Variance:  $α=E[X^{2}]-E[X]^{2}=p-p^{2}$


Characteristics of Bernoulli Distribution

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Unbiased Estimator for Bernoulli Parameter

See this [youtube video](https://www.youtube.com/watch?v=D1hgiAla3KI) for a explanation for why 
$$\mathbb{E}[\hat{\sigma}^2_m] =\mathbb{E}\left[\frac{1}{m - 1}\sum_{i=1}^m \left(x^{(i)} - \hat{\mu}_m\right)^2\right]$$
is an unbiased estimator for the variance $\sigma^2$. 

Unbiased Estimator for Normal $\sigma^2$ parameter

When we have $m$ $i.i.d$ samples  $\{x^{(1)},...,x^{(m)}\}$ from a Normal distribution with parameter $\mu$ and $\sigma$, such that: 
	$$P(x^{(i)};\mu, \sigma) = \frac{1}{\sigma\sqrt{2\pi}}e^{\frac{1}{2}(\frac{x-\mu}{\sigma})^2}$$,
a unbiased estimator for $\mu$ is the mean of those samples:
	$\hat{\mu}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$


Unbiased Estimator for Normal $\mu$ Parameter

A “Bernoulli trial” is an experiment or case where the outcome follows a Bernoulli distribution. Some common examples of Bernoulli trials include:

1) A single flip of a coin that may have a head(0) or a tail(1) outcome.

2) A single birth of either a boy(0) or a girl(1).

Learn Before

Related

Learn After