Training a neural network with a small dataset can cause the network to memorize all training examples, in turn leading to overfitting and poor performance on a holdout dataset. Adding noise to a neural network during training can improve the robustness of the network, resulting in better generalization and faster learning. 


Noise is traditionally added to the inputs, but can also be added to weights, gradients, and even activation functions. It is important to remember that noise injection can be much more powerful than simply shrinking the parameters, especially when the noise is added to the hidden units. 

Noise Robustness

- To improve the robustness of the neural network in speech recognition, noise injection is a method we can apply by adding random noise in the input to the neural network. One of the examples is to inject Gaussian noise into the input. 
- Applying noise injection to the hidden units can be highly effective when we tune the magnitude of the noise carefully. 

University of Michigan - Ann Arbor

Getting more data may be costly and an ineffective way to prevent overfitting. However, you can augment your data to create more. We are adding synthetic data modified from our original data set. You essentially transform your data in a way where it is different form the original, but the data still fits into your given class. It is much more efficient and cost effective. 

Data Augmentation in Deep Learning

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

For example, instead of taking more cat pictures, you can use one or a combination of the following methods:
- Mirroring (Horizontally/vertically flipping)
- Random cropping (Zooming in)
- Rotating
- Shearing
- Local wrapping
- Color shifting

Learn Before

Related

Learn After