Adding noise to an under constrained neural network model with a small training dataset can have a regularizing effect and reduce overfitting. The noise can be used in a few different ways with a neural network model. 

- Firstly, it can be used as an input layer to add noise to input variables directly. 
- Secondly, noise can also be added between hidden layers in the model. Given the flexibility of Keras, the noise can be added before or after the use of the activation function. 
- Noise can also be used for MLP Noise Regularization, CNN Noise Regularization and RNN Dropout Regularization. Here are some code examples: 

https://machinelearningmastery.com/how-to-improve-deep-learning-model-robustness-by-adding-noise/

University of Michigan - Ann Arbor

Training a neural network with a small dataset can cause the network to memorize all training examples, in turn leading to overfitting and poor performance on a holdout dataset. Adding noise to a neural network during training can improve the robustness of the network, resulting in better generalization and faster learning. 


Noise is traditionally added to the inputs, but can also be added to weights, gradients, and even activation functions. It is important to remember that noise injection can be much more powerful than simply shrinking the parameters, especially when the noise is added to the hidden units. 

Noise Robustness

Gaussian Noise is statistical noise having a Probability Density Function (PDF) equal to that of the normal distribution. It is also known as the Gaussian Distribution.

For instance, if we observe the red PDF: the mean value of the noise will be -2. So, on average, 2 would be subtracted from all pixels of the image. However, if we observe the orange PDF, the mean value is 3. So on average, 3 would be added to all pixels. 

Gaussian Noise

When adding a Gaussian Noise layer as the input layer,  we are creating more samples and making the data distribution smoother. 

It can be seen on the line graph that noise causes the accuracy and loss of the model to jump around due to the points with noise that have been introduced in the training and that conflict with the points of the training data set. The std used here is 0.1 as input noise and which might be a little bit high. In this case, the overfitting in this model has been decreased, furthermore,  the test accuracy has been improved to from 0.585 to 0.642.

Learn Before

Related