- Vanishing Gradient Problem
- Zero Centered
- Computational Expense
- Differentiable

University of Minnesota - Twin Cities

In simple terms, an activation function is added in the neural network in order to help the network learn complex patterns in the data. When comparing this to the brain a neuron sends signals to other neurons but what signals being sent depends on the conditions. The activation function is at the end taking in the data from the node and transforms it into an input that a new node (neuron) can take into the new layer.

- And of course, typically math is involved with coming up with these functions.

Activation Functions

Desired Features of an Activation Function

Luckily it can be categorized into 2 types:

- Linear Activation Function

- Non-linear Activation Function

Learn Before

Related