Why do we normalize the inputs X in Deep Learning?

In one hypothesis, it helps to eliminate internal covariate shifts.

Visualization of How Batch Norm Helps Speed up Deep Learning

Since we normalize the inputs in each layer, the changes in the previous layer won't have much influence on the next layer. This is because, it reduces the amount of shift in the distribution of inputs to each layer (covariate shifts).

Batch Norm Makes Training Deep Networks Easier

Batch Norm has a slight regularization effect. Since we shift and scale *z* based on the mean and variance of only a single batch, it adds some noise into each hidden layer activation. So like dropout, it will add noise to each hidden layer’s activation. However, the effect is so small that we might want to use it together with other regularization methods like dropout.
If one uses a larger mini-batch size, it'd create less noise and thereby reduce the regularization effect. 

Batch Norm as Regularization in Deep Learning

- It makes your hyperparameter search problem much easier.
- It makes the neural network more robust to the choice of hyperparameters in a much bigger range of hyperparameters.
- It makes training even very deep networks easier.
- It has a slight regularization effect.

University of Michigan - Ann Arbor

Batch normalization is a method that helps to speed neural network training. The basic idea is that we shift and scale the inputs to every layer of neural network. 
For example, for a 3-layer neural network shown below, we want to normalize $z^{[3]}$ to speed the training of $w^{[3]}$ and $b^{[3]}$ in the third layer.

Batch Normalization in Deep Learning

https://www.coursera.org/learn/deep-neural-network?specialization=deep-learning

Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization

Here is a useful website about Batch Norm: https://www.cerebras.net/neurips-2019-online-normalization-for-training-neural-networks/

NeurIPS 2019: Online Normalization for Training Neural Networks

Benefits of Batch Norm in Deep Learning

In each layer, before we put our input $z^{(i)}$ to the activation function, we first normalize it by Batch Norm.  That is,  we first calculate 
$\mu = \frac{1}{n}\sum_{i} z^{(i)}$
$\sigma^2=\frac{1}{n}\sum_{i}(z^{(i)}-\mu)^2$
$z_{norm}^{(i)} = \frac{z^{(i)}-\mu}{\sqrt{\sigma^2+\epsilon}}$
$\tilde{z}^{(i)}=\gamma^{(i)} z_{norm}^{(i)} + \beta^{(i)}$
We introduce $\gamma^{(i)}$ and $\beta^{(i)}$ because we don't want all the inputs to neurons in hidden layers to always have mean 0 and variance 1.
$\Rightarrow a^{(i)} = g(\tilde{z}^{(i)})$,
where g is some activation function.
Recall in each layer,
$z^{(i)}= W^{(i)} a^{(i-1)}+b^{(i)}$
When we calculate $\tilde{z}^{(i)}$, we first normalize $z^{(i)}$ by subtracting the mean. So the value of $b^{(i)}$ has no influence on the result. $\Rightarrow$ In each layer, we have three parameters  $W^{(i)}, \gamma^{(i)},  \beta^{(i)}$.
Frequently, we combine this with mini-batches; i.e., we train $W^{(i)}$, $\gamma^{(i)}$,  $\beta^{(i)}$ with respect to $X^{\{1\}}, ..., X^{\{n\}}$, where $X^{\{i\}}$ is $i^{th}$ batch of the training data.

Learn Before

Related

Learn After