We estimate $\mu, \sigma^2$ using exponentially weighted average across mini-batches during the training phase and we use the estimated values for the calculation of $Z_{norm}$ in the test time.

University of Michigan - Ann Arbor

In each layer, before we put our input $z^{(i)}$ to the activation function, we first normalize it by Batch Norm.  That is,  we first calculate 
$\mu = \frac{1}{n}\sum_{i} z^{(i)}$
$\sigma^2=\frac{1}{n}\sum_{i}(z^{(i)}-\mu)^2$
$z_{norm}^{(i)} = \frac{z^{(i)}-\mu}{\sqrt{\sigma^2+\epsilon}}$
$\tilde{z}^{(i)}=\gamma^{(i)} z_{norm}^{(i)} + \beta^{(i)}$
We introduce $\gamma^{(i)}$ and $\beta^{(i)}$ because we don't want all the inputs to neurons in hidden layers to always have mean 0 and variance 1.
$\Rightarrow a^{(i)} = g(\tilde{z}^{(i)})$,
where g is some activation function.
Recall in each layer,
$z^{(i)}= W^{(i)} a^{(i-1)}+b^{(i)}$
When we calculate $\tilde{z}^{(i)}$, we first normalize $z^{(i)}$ by subtracting the mean. So the value of $b^{(i)}$ has no influence on the result. $\Rightarrow$ In each layer, we have three parameters  $W^{(i)}, \gamma^{(i)},  \beta^{(i)}$.
Frequently, we combine this with mini-batches; i.e., we train $W^{(i)}$, $\gamma^{(i)}$,  $\beta^{(i)}$ with respect to $X^{\{1\}}, ..., X^{\{n\}}$, where $X^{\{i\}}$ is $i^{th}$ batch of the training data.

Batch Norm in Deep Learning Implementation

For t=1,..., n mini-batches,
         Compute forward propagation on $X^{\{i\}}$
                 In each hidden layer, use Batch Norm to replace $z^{(l)}$ with $\tilde{z}^{(l)}$
          Use back propagation to compute $dW^{(l)}, d\gamma^{(l)},  d\beta^{(l)}$
          Update parameters:
                   $W^{(l)} = W^{(l)} - dW^{(l)}$
                   $\gamma^{(l)} = \gamma^{(l)} - d\gamma^{(l)}$
                   $\beta^{(l)} = \beta^{(l)} - d\beta^{(l)}$

Learn Before

Related