$$g(x) = Max(0.1x, x) \Rightarrow g'(x) =  \left\{ \begin{array}{rl} 0.1 & if x < 0 \\ 1 & if x \geq 0 \end{array}\right.$$ 

University of Michigan - Ann Arbor

Leaky ReLU

Pros:
 - Prevents dying ReLU problem—this variation of ReLU has a small positive slope in the negative area, so it does enable backpropagation, even for negative input values
 - Computationally efficient—allows the network to converge very quickly
 - Non-linear—although it looks like a linear function, Leakly ReLU has a derivative function and allows for backpropagation.

Cons:
- Results not consistent—leaky ReLU does not provide consistent predictions for negative input values.

Learn Before

Related