Here is a useful website about Batch Norm: https://www.cerebras.net/neurips-2019-online-normalization-for-training-neural-networks/

University of Michigan - Ann Arbor

Batch normalization is a method that helps to speed neural network training. The basic idea is that we shift and scale the inputs to every layer of neural network. 
For example, for a 3-layer neural network shown below, we want to normalize $z^{[3]}$ to speed the training of $w^{[3]}$ and $b^{[3]}$ in the third layer.

Batch Normalization in Deep Learning

NeurIPS 2019: Online Normalization for Training Neural Networks

- It makes your hyperparameter search problem much easier.
- It makes the neural network more robust to the choice of hyperparameters in a much bigger range of hyperparameters.
- It makes training even very deep networks easier.
- It has a slight regularization effect.

Benefits of Batch Norm in Deep Learning

In each layer, before we put our input $z^{(i)}$ to the activation function, we first normalize it by Batch Norm.  That is,  we first calculate 
$\mu = \frac{1}{n}\sum_{i} z^{(i)}$
$\sigma^2=\frac{1}{n}\sum_{i}(z^{(i)}-\mu)^2$
$z_{norm}^{(i)} = \frac{z^{(i)}-\mu}{\sqrt{\sigma^2+\epsilon}}$
$\tilde{z}^{(i)}=\gamma^{(i)} z_{norm}^{(i)} + \beta^{(i)}$
We introduce $\gamma^{(i)}$ and $\beta^{(i)}$ because we don't want all the inputs to neurons in hidden layers to always have mean 0 and variance 1.
$\Rightarrow a^{(i)} = g(\tilde{z}^{(i)})$,
where g is some activation function.
Recall in each layer,
$z^{(i)}= W^{(i)} a^{(i-1)}+b^{(i)}$
When we calculate $\tilde{z}^{(i)}$, we first normalize $z^{(i)}$ by subtracting the mean. So the value of $b^{(i)}$ has no influence on the result. $\Rightarrow$ In each layer, we have three parameters  $W^{(i)}, \gamma^{(i)},  \beta^{(i)}$.
Frequently, we combine this with mini-batches; i.e., we train $W^{(i)}$, $\gamma^{(i)}$,  $\beta^{(i)}$ with respect to $X^{\{1\}}, ..., X^{\{n\}}$, where $X^{\{i\}}$ is $i^{th}$ batch of the training data.

Learn Before

Related