The numerical output. ofa sigmoid node in a neural netork

University of Michigan - Ann Arbor

$S(x) = \frac{1}{1 +e^{-x}} = \frac{e^x}{e^x + 1}$ 


Sigmoid/Logistic Function

Pros:
 - Smooth gradient, preventing “jumps” in output values.
 - Output values bound between 0 and 1, normalizing the output of each neuron.
 - Clear predictions—For X above 2 or below -2, tends to bring the Y value (the prediction) to the edge of the curve, very close to 1 or 0. This enables clear predictions.

Cons:
 - Vanishing gradient—for very high or very low values of X, there is almost no change to the prediction, causing a vanishing gradient problem. This can result in the network refusing to learn further, or being too slow to reach an accurate prediction.
 - Outputs not zero centered.
 - Computationally expensive


Pros and Cons of Sigmoid/Logistic Function

$$g(x) = \frac{1}{1 + e^{-x}} \Rightarrow g'(x) = \frac{d}{dx}g(x) = \frac{1}{1 + e^{-x}} (1 - \frac{1}{1 + e^{-x}}) = g(x) (1 - g(x))$$ 

Derivative of Sigmoid/Logistic Function

- tanh works almost always better than Sigmoid function, but Sigmoid should be used for the output layer of binary classification models.
- tanh is in range (-1, 1), but Sigmoid is in range (0, 1).
- Using tanh activation functions in the hidden layers optimizes training because it centers the data around 0 rather tahn 0.5

Learn Before

Related