In one hypothesis, it helps to eliminate internal covariate shifts.

University of Michigan - Ann Arbor

- It makes your hyperparameter search problem much easier.
- It makes the neural network more robust to the choice of hyperparameters in a much bigger range of hyperparameters.
- It makes training even very deep networks easier.
- It has a slight regularization effect.

Benefits of Batch Norm in Deep Learning

Here is a useful website about Batch Norm: https://www.cerebras.net/neurips-2019-online-normalization-for-training-neural-networks/

NeurIPS 2019: Online Normalization for Training Neural Networks

Why do we normalize the inputs X in Deep Learning?

Visualization of How Batch Norm Helps Speed up Deep Learning

Since we normalize the inputs in each layer, the changes in the previous layer won't have much influence on the next layer. This is because, it reduces the amount of shift in the distribution of inputs to each layer (covariate shifts).

Batch Norm Makes Training Deep Networks Easier

Batch Norm has a slight regularization effect. Since we shift and scale *z* based on the mean and variance of only a single batch, it adds some noise into each hidden layer activation. So like dropout, it will add noise to each hidden layer’s activation. However, the effect is so small that we might want to use it together with other regularization methods like dropout.
If one uses a larger mini-batch size, it'd create less noise and thereby reduce the regularization effect. 

Learn Before

Related