When implementing dropout in a neural network, the dropout operation is typically applied to the output of each hidden layer immediately following its non-linear activation function. This ensures that the neurons randomly zeroed out are the activated representations of the layer's output.

Claude

When dropout is applied to a hidden layer of a neural network, each hidden unit is zeroed out with a specified probability $$p$$. This effectively creates a sub-network containing only a subset of the original neurons. Consequently, the forward calculation of the outputs and the backward calculation of gradients during backpropagation no longer depend on the dropped nodes. This mechanism ensures that the output layer cannot become overly dependent on any single hidden unit.

Mechanics of Dropout on a Hidden Layer

Dive into Deep Learning

To implement dropout for a layer computationally, we must draw samples from a Bernoulli distribution for each dimension, where a node is kept (value $$1$$) with probability $$1-p$$ and dropped (value $$0$$) with probability $$p$$. An efficient way to achieve this is to generate a tensor of samples from a uniform distribution $$U[0, 1]$$ and apply a threshold, keeping only those elements where the uniformly sampled value is strictly greater than the dropout probability $$p$$.

Implementing Dropout via Uniform Sampling

Applying Dropout After Activation

In a neural network architecture, dropout probabilities do not have to be uniform; they can be configured independently for each hidden layer. A common heuristic is to set a lower dropout probability for the layers closer to the input, as these initial layers often capture fundamental, low-level features that are critical for the network's overall performance.

Learn Before

Related