Activation functions are a critical part of the design of a neural network- A hidden layer in a neural network is a layer that receives input from another layer (such as another hidden layer or an input layer) and provides output to another layer (such as another hidden layer or an output layer).

There are perhaps three activation functions you may want to consider for use in hidden layers; they are:

• Rectified Linear Activation (ReLU) The rectified linear activation function calculated as max(0.0, x) It is common because it is both simple to implement and effective at overcoming the limitations of other previously popular activation functions, such as Sigmoid and Tanh.

• Logistic (Sigmoid) The sigmoid activation function is also called the logistic function, and calculated as 1.0 / (1.0 + e^-x) It is used in the logistic regression classification algorithm.

• Hyperbolic Tangent (Tanh) The function takes any real value as input and outputs values in the range -1 to 1. The larger the input (more positive), the closer the output value will be to 1.0, whereas the smaller the input (more negative), the closer the output will be to -1.0. IT is calculated as (e^x – e^-x) / (e^x + e^-x)

It is common practive to choose activation function based on the type of model you work on. Multilayer Perceptron (MLP): ReLU activation function. Convolutional Neural Network (CNN): ReLU activation function. Recurrent Neural Network: Tanh and/or Sigmoid activation function.