https://jamesmccaffrey.wordpress.com/2019/05/07/neural-network-dropout-and-inverted-dropout/

University of Michigan - Ann Arbor

Unlike L2 or L1 regularization, Dropout regularization works by assigning a probability P, which is the probability that a given node is turned off for the current iteration of training. By doing this, you make the model simpler while training, which will reduce overfitting, while still gaining the benefits a larger model has during testing. Also, no single node will have excess influence on the model, and the model will spread the weights out, giving us a similar effect to L2 regularization. Generally, Dropout Regularization is the preferred form of regularization, as it not only accomplishes results similar to L2 regularization, it also provides some robustness to the model as each iteration is randomized.

Dropout Regularization in Deep Learning

It is successful because:
- With each iteration, you are working with a smaller network
- It will tend to shrink the squared norm of the weights
- Shrinking these weights prevents overfitting

Why does dropout work?

Neural Network Dropout and Inverted Dropout

In this version of dropout regularization, the activation vector is not only multiplied by a 0-1 dropout vector but also divided by the probability P (see parent node). This additional division, which is done at training time, preserves the expected value of the activations. This makes test time faster by eliminating scaling issues before the test phase.

Inverted Dropout Technique

  - The cost function isn't well-defined.
  - The cost can be really significant in the case of a complete system to the point where computational cost can outweigh its normal benefits
  - Dropout is also less effective when extremely few labeled training examples are available

Disadvantages of Dropout

For the variable P, as P gets larger, the regularization will decrease and the training error will be lower. Often times you will set P = 1 for certain layers that aren't overfitting as much in order to ensure you are keeping every unit in that layer.

Learn Before

Related