- Sigmoid or Logistic Activation Function
- Tanh Activation Function
- ReLU (Rectified Linear Unit) Activation Function
- Softmax Activation Function

Non-Linear Activation Functions

https://www.nbshare.io/notebook/751082217/Activation-Functions-In-Python/

More Activation Functions

The identity function is the only function belonging to the linear activation-function type. It is defined as $$f(x) = x$$, so the output equals the input unchanged. Unlike non-linear activation functions such as sigmoid or tanh, its output is not compressed into a bounded range.

Linear Activation Function - Identity

Luckily it can be categorized into 2 types:

- Linear Activation Function

- Non-linear Activation Function

University of Minnesota - Twin Cities

In simple terms, an activation function is added in the neural network in order to help the network learn complex patterns in the data. When comparing this to the brain a neuron sends signals to other neurons but what signals being sent depends on the conditions. The activation function is at the end taking in the data from the node and transforms it into an input that a new node (neuron) can take into the new layer.

- And of course, typically math is involved with coming up with these functions.

Activation Functions

- Vanishing Gradient Problem
- Zero Centered
- Computational Expense
- Differentiable

Learn Before

Related

Learn After

Learn After

Related