When evaluating a dropout layer implementation, its behavior varies predictably at boundary probabilities. Applying a dropout operation with a probability of $$0$$ leaves the input completely unchanged, as no elements are dropped. Using a dropout probability of $$0.5$$ drops approximately half of the elements and rescales the surviving ones by a factor of $$2$$. Conversely, applying a dropout probability of $$1$$ drops all elements entirely, resulting in a tensor of zeros.

Dropout Behavior at Extreme Probabilities

A dropout layer can be implemented algorithmically from scratch by first checking if the dropout probability $$p$$ equals $$1$$; if so, it immediately returns a tensor of zeros. For $$p < 1$$, it generates a boolean mask by evaluating where a tensor of uniform random numbers $$U[0, 1]$$ is strictly greater than $$p$$. This boolean mask is cast to a floating-point format, multiplied element-wise with the input tensor, and divided by $$1 - p$$ to rescale the surviving elements.

Claude

To implement dropout for a layer computationally, we must draw samples from a Bernoulli distribution for each dimension, where a node is kept (value $$1$$) with probability $$1-p$$ and dropped (value $$0$$) with probability $$p$$. An efficient way to achieve this is to generate a tensor of samples from a uniform distribution $$U[0, 1]$$ and apply a threshold, keeping only those elements where the uniformly sampled value is strictly greater than the dropout probability $$p$$.

Implementing Dropout via Uniform Sampling

In modern implementations of dropout (often called inverted dropout), the activation tensor is not only multiplied by a binary mask but the remaining values are also rescaled. If elements are dropped out with probability $$p$$, the surviving elements are divided by $$1 - p$$. This rescaling step, performed during training, preserves the expected value of the activations and eliminates the need for scaling adjustments during the test phase.

Learn Before

Related

Learn After