$$g(x) = \frac{1}{1 + e^{-x}} \Rightarrow g'(x) = \frac{d}{dx}g(x) = \frac{1}{1 + e^{-x}} (1 - \frac{1}{1 + e^{-x}}) = g(x) (1 - g(x))$$ 

University of Michigan - Ann Arbor

$S(x) = \frac{1}{1 +e^{-x}} = \frac{e^x}{e^x + 1}$ 


Sigmoid/Logistic Function

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

Pros:
 - Smooth gradient, preventing “jumps” in output values.
 - Output values bound between 0 and 1, normalizing the output of each neuron.
 - Clear predictions—For X above 2 or below -2, tends to bring the Y value (the prediction) to the edge of the curve, very close to 1 or 0. This enables clear predictions.

Cons:
 - Vanishing gradient—for very high or very low values of X, there is almost no change to the prediction, causing a vanishing gradient problem. This can result in the network refusing to learn further, or being too slow to reach an accurate prediction.
 - Outputs not zero centered.
 - Computationally expensive


Pros and Cons of Sigmoid/Logistic Function

Derivative of Sigmoid/Logistic Function

The numerical output. ofa sigmoid node in a neural netork

Question about sigmoid node

- The hyperbolic tangent (tanh) function outputs values in the range $$(-1, 1)$$, while the sigmoid function outputs values in the range $$(0, 1)$$.
- tanh generally works better than sigmoid as a hidden-layer activation function, because its output is centered around 0 rather than 0.5, which tends to make training/optimization faster.
- Sigmoid is still preferred for the output layer of binary classification models, since its $$(0, 1)$$ range maps directly to a class probability.

Learn Before

Related