University of Michigan - Ann Arbor

- It makes your hyperparameter search problem much easier.
- It makes the neural network more robust to the choice of hyperparameters in a much bigger range of hyperparameters.
- It makes training even very deep networks easier.
- It has a slight regularization effect.

Benefits of Batch Norm in Deep Learning

Why do we normalize the inputs X in Deep Learning?

In one hypothesis, it helps to eliminate internal covariate shifts.

Visualization of How Batch Norm Helps Speed up Deep Learning

Since we normalize the inputs in each layer, the changes in the previous layer won't have much influence on the next layer. This is because, it reduces the amount of shift in the distribution of inputs to each layer (covariate shifts).

Batch Norm Makes Training Deep Networks Easier

Batch Norm has a slight regularization effect. Since we shift and scale *z* based on the mean and variance of only a single batch, it adds some noise into each hidden layer activation. So like dropout, it will add noise to each hidden layer’s activation. However, the effect is so small that we might want to use it together with other regularization methods like dropout.
If one uses a larger mini-batch size, it'd create less noise and thereby reduce the regularization effect. 

Learn Before

Related