Training a neural network with a small dataset can cause the network to memorize all training examples, in turn leading to overfitting and poor performance on a holdout dataset. Adding noise to a neural network during training can improve the robustness of the network, resulting in better generalization and faster learning.

Noise is traditionally added to the inputs, but can also be added to weights, gradients, and even activation functions. It is important to remember that noise injection can be much more powerful than simply shrinking the parameters, especially when the noise is added to the hidden units.