$$g(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} \Rightarrow g'(x) = \frac{d}{dx}g(x) = 1 - (g(x))^{2}$$ 

University of Michigan - Ann Arbor

$S(x)  = tanh(x) = \frac{e^x-e^{-x}}{e^x + e^{-x}}$


TanH/Hyperbolic Tangent Function

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0

More on the Tanh function

Pros:
 - Zero centered—making it easier to model inputs that have strongly negative, neutral, and strongly positive values.
 - Smooth gradient, preventing “jumps” in output values.
 - Output values bound between -1 and 1, normalizing the output of each neuron.
 - Clear predictions—For X above 2 or below -2, tends to bring the Y value (the prediction) to the edge of the curve, very close to 1 or -1. This enables clear predictions.

Cons (Like the Sigmoid function):
 - Vanishing gradient—for very high or very low values of X, there is almost no change to the prediction, causing a vanishing gradient problem. This can result in the network refusing to learn further, or being too slow to reach an accurate prediction.
 - Computationally expensive

Pros and Cons of Hyperbolic Tangent Function

You have built a network using the tanh activation for all the hidden units. You initialize the weights to relative large values, using np.random.randn(..,..)*1000. What will happen?

- The hyperbolic tangent (tanh) function outputs values in the range $$(-1, 1)$$, while the sigmoid function outputs values in the range $$(0, 1)$$.
- tanh generally works better than sigmoid as a hidden-layer activation function, because its output is centered around 0 rather than 0.5, which tends to make training/optimization faster.
- Sigmoid is still preferred for the output layer of binary classification models, since its $$(0, 1)$$ range maps directly to a class probability.

Learn Before

Related

Related