https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0

University of Michigan - Ann Arbor

$S(x)  = tanh(x) = \frac{e^x-e^{-x}}{e^x + e^{-x}}$


TanH/Hyperbolic Tangent Function

Pros:
 - Zero centered—making it easier to model inputs that have strongly negative, neutral, and strongly positive values.
 - Smooth gradient, preventing “jumps” in output values.
 - Output values bound between -1 and 1, normalizing the output of each neuron.
 - Clear predictions—For X above 2 or below -2, tends to bring the Y value (the prediction) to the edge of the curve, very close to 1 or -1. This enables clear predictions.

Cons (Like the Sigmoid function):
 - Vanishing gradient—for very high or very low values of X, there is almost no change to the prediction, causing a vanishing gradient problem. This can result in the network refusing to learn further, or being too slow to reach an accurate prediction.
 - Computationally expensive

Pros and Cons of Hyperbolic Tangent Function

$$g(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} \Rightarrow g'(x) = \frac{d}{dx}g(x) = 1 - (g(x))^{2}$$ 

Derivative of TanH/Hyperbolic Tangent Function

More on the Tanh function

- tanh works almost always better than Sigmoid function, but Sigmoid should be used for the output layer of binary classification models.
- tanh is in range (-1, 1), but Sigmoid is in range (0, 1).
- Using tanh activation functions in the hidden layers optimizes training because it centers the data around 0 rather tahn 0.5

Sigmoid/Logistic vs. TanH/Hyperbolic Tangent functions

You have built a network using the tanh activation for all the hidden units. You initialize the weights to relative large values, using np.random.randn(..,..)*1000. What will happen?

Learn Before

Related