Concept

Batch Norm as Regularization in Deep Learning

Batch Norm has a slight regularization effect. Since we shift and scale z based on the mean and variance of only a single batch, it adds some noise into each hidden layer activation. So like dropout, it will add noise to each hidden layer’s activation. However, the effect is so small that we might want to use it together with other regularization methods like dropout. If one uses a larger mini-batch size, it'd create less noise and thereby reduce the regularization effect.

0

2

Updated 2021-03-31

Tags

Data Science