$$\textrm{bias}(\hat{\theta}_m) = \mathbb{E}[\hat{\theta}_m] - \theta \\
= \mathbb{E}\left[\frac{1}{m} \sum_{i=1}^{m}x^{(i)}\right] - \theta \\
	= \frac{1}{m} \sum_{i=1}^{m}\mathbb{E}\left[x^{(i)}\right] - \theta \\
	= \frac{1}{m} \sum_{i=1}^{m}\sum_{x^{(i)} = 0}^1\left( x^{(i)}\theta^{x^{(i)}}(1-\theta)^{(1-x^{(i)})}\right)- \theta \\
	= \frac{1}{m} \sum_{i=1}^m(\theta) - \theta \\
	= \theta - \theta = 0$$

Reasons that the steps above work:  The first one is the definition of bias. The second one is plugging in our estimator. The third one is because the expected value of the sum is the sum of the expected values. The fourth is by the definition of expected value of a discrete random variable (it is each value times the probability at that value, which for a bernoulli r.v. is $\theta$ if $x = 1$, and $1 - \theta$ if $x = 0$, which can be summarized as $P(x;\theta) = \theta^x(1-\theta)^{1-x}$. The fifth line works because we have assumed identically distributed random variables (note that we also assumed that they are independent, but this isn’t necessary: the same derivation holds when the random variables are dependent). Then getting from the fifth to the sixth lines requires recognizing that the sum of $m$ $\theta$'s is equal to $m*\theta$, and when you multiply that by $\frac{1}{m}$ you get $\theta$. 

Proof that the Mean of Bernoulli samples is an Unbiased Estimator for the Bernoulli Parameter $\theta$ 

When we have $$m$$ i.i.d. samples $$\left\{x^{(1)}, \dots, x^{(m)}\right\}$$ from a Bernoulli distribution with parameter $$\theta$$, such that $$P(x^{(i)},\theta) = \theta^{x^{(i)}} (1-\theta)^{(1-x^{(i)})}$$, an unbiased estimator for $$\theta$$ is the mean of those samples:

$$\hat{\theta}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$$.

Johns Hopkins University

Google

When we take $$n$$ i.i.d. samples from a Bernoulli distribution with parameter $$\theta$$, the mean of those samples is an unbiased estimator for $$\theta$$. Similarly, the mean of $$n$$ i.i.d. samples from a Normal distribution is an unbiased estimator for the $$\mu$$ parameter of the Normal distribution.

Examples of Unbiased Estimators

For a Bernoulli distribution characterizing a random variable $$ X $$ with probability $$ p $$ for $$ X=1 $$ ("success") and $$ 1-p $$ for $$ X=0 $$ ("failure"), the key characteristics are:
1) Probability mass function: $$ P(X) = p^{X}(1-p)^{1-X} $$
2) Expectation: $$ E[X] = p(1) + (1-p)(0) = p $$
3) Variance: $$ Var(X) = E[X^2] - (E[X])^2 = p - p^2 = p(1-p) $$

Characteristics of Bernoulli Distribution

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

An unbiased estimator for the variance $$\sigma^2$$ of a Normal distribution can be demonstrated through the expected value of the sample variance: $$\mathbb{E}[\hat{\sigma}^2_m] =\mathbb{E}\left[\frac{1}{m - 1}\sum_{i=1}^m \left(x^{(i)} - \hat{\mu}_m\right)^2\right]$$. See this [YouTube video](https://www.youtube.com/watch?v=D1hgiAla3KI) for a detailed explanation of this derivation.

Unbiased Estimator for Normal $$\sigma^2$$ Parameter

When we have $$m$$ i.i.d. samples $$\left\{x^{(1)},...,x^{(m)}\right\}$$ from a Normal distribution with parameters $$\mu$$ and $$\sigma$$, such that: $$p(x^{(i)};\mu, \sigma) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{1}{2\sigma^2}\left(x^{(i)}-\mu\right)^2\right)$$, an unbiased estimator for $$\mu$$ is the sample mean: $$\hat{\mu}_m = \frac{1}{m} \sum_{i=1}^m x^{(i)}$$.

Learn Before

Related

Learn After