Non-linear functions address the problems of a linear activation function:

 - They allow backpropagation because they have a derivative function which is related to the inputs.
 - They allow “stacking” of multiple layers of neurons to create a deep neural network. Multiple hidden layers of neurons are needed to learn complex data sets with high levels of accuracy.

Linear vs. Non-Linear Activation Functions

$S(x) = \frac{1}{1 +e^{-x}} = \frac{e^x}{e^x + 1}$ 


Sigmoid/Logistic Function

$S(x)  = tanh(x) = \frac{e^x-e^{-x}}{e^x + e^{-x}}$


TanH/Hyperbolic Tangent Function

The Swish function, introduced by Ramachandran et al. in 2017, is a mathematical function defined as the product of its input and the sigmoid function applied to a scaled version of the input. For a scalar input $x$, it is expressed as:

$ \operatorname{swish}(x) = x \cdot \operatorname{sigmoid}(\beta x) = \frac{x}{1+e^{-\beta x}}$

where $\beta$ is a constant or a trainable parameter. When applied element-wise to a vector $\mathbf{h}$ in neural networks, the formula is written as:

$\sigma_{\text{swish}}(\mathbf{h}) = \mathbf{h} \odot \text{Sigmoid}(\beta \mathbf{h})$

where $\odot$ denotes the element-wise product.

Swish Function

The Rectified Linear Unit (ReLU) is a common choice for the activation function $$\sigma(\cdot)$$ within the hidden layers of neural networks. It is defined to output the positive portion of its argument. When applied to an input vector $$\mathbf{h}$$, the ReLU function is given by the formula: $$\sigma_{\mathrm{relu}}(\mathbf{h}) = \mathrm{max}(0, \mathbf{h})$$.

ReLU (Rectified Linear Unit)

 If x > 0:
 $f(x)=x$
 If x $<=$  1:
$f(x) = α(e^x - 1)$

ELU (Exponential Linear Unit)

Which activation function is represented by each of these plots?

Which of the following introduces nonlinearity into neural networks?

To convert raw, unnormalized outputs $$\mathbf{o}$$ into valid probabilities, the softmax function applies an exponential function to each component and then normalizes them by their sum. The exponentiation ensures that all probabilities are non-negative, while the division ensures that they sum to $$1$$. Mathematically, the predicted probability distribution $$\hat{\mathbf{y}}$$ is defined as:

$$\hat{\mathbf{y}} = \mathrm{softmax}(\mathbf{o}) \quad 	extrm{where}\quad \hat{y}_i = \frac{\exp(o_i)}{\sum_j \exp(o_j)}$$

This guarantees that $$0 \le \hat{y}_i \le 1$$ and $$\sum_j \hat{y}_j = 1$$. Unlike other normalizations or the probit model, the softmax function preserves order and leads to a more well-behaved optimization problem.

Softmax Function

Modern neural network models use non-linear activation functions. They allow the model to create complex mappings between the network’s inputs and outputs, which are essential for learning and modeling complex data, such as images, video, audio, and data sets which are non-linear or have high dimensionality.

There are 8 common non-linear activation functions:
 - Sigmoid / Logistic
 - TanH / Hyperbolic Tangent
 - ELU (Exponential Linear Unit)
 - ReLU (Rectified Linear Unit)
 - Leaky ReLU
 - Parametric ReLU
 - Softmax
 - Swish

University of Michigan - Ann Arbor

There are 12 types of activation functions:

- Binary Step
- Linear
- Sigmoid
- Tanh
- ReLU
- Leaky ReLU
- Parameterised ReLU
- Exponential Linear Unit
- Swish
- Softmax

https://www.analyticsvidhya.com/blog/2020/01/fundamentals-deep-learning-activation-functions-when-to-use-them/

Types of Activation Functions

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

A binary step function is a threshold-based activation function. If the input value is above or below a certain threshold, the neuron is activated and sends exactly the same signal to the next layer.

Binary Step Function

A linear activation function takes the form: $y = cx$.

It takes the inputs, multiplied by the weights for each neuron, and creates an output signal proportional to the input. In one sense, a linear function is better than a step function because it allows multiple outputs, not just yes and no.


Linear Activation Function

Non-Linear Activation Functions

Sigmoid 
Pros: useful for binary classification, output layer 
Cons: shifted from center

Tanh 
Pros: has centering effect, almost always superior to sigmoid 
Cons: complex in comparison to ReLU 

ReLU
Pros: simpler, default
Cons: Derivative is 0 when z is negative --> leaky reLU

Pros and Cons of each Activation Function

Activation functions are a critical part of the design of a neural network- A hidden layer in a neural network is a layer that receives input from another layer (such as another hidden layer or an input layer) and provides output to another layer (such as another hidden layer or an output layer).

There are perhaps three activation functions you may want to consider for use in hidden layers; they are:

- Rectified Linear Activation (ReLU)
The rectified linear activation function calculated as max(0.0, x)
It is common because it is both simple to implement and effective at overcoming the limitations of other previously popular activation functions, such as Sigmoid and Tanh.


- Logistic (Sigmoid)
The sigmoid activation function is also called the logistic function, and calculated as 1.0 / (1.0 + e^-x)
It is used in the logistic regression classification algorithm.

- Hyperbolic Tangent (Tanh)
The function takes any real value as input and outputs values in the range -1 to 1. The larger the input (more positive), the closer the output value will be to 1.0, whereas the smaller the input (more negative), the closer the output will be to -1.0.
IT is calculated as (e^x – e^-x) / (e^x + e^-x)

It is common practive to choose activation function based on the type of model you work on.
Multilayer Perceptron (MLP): ReLU activation function.
Convolutional Neural Network (CNN): ReLU activation function.
Recurrent Neural Network: Tanh and/or Sigmoid activation function.



How to Choose an Activation Function for Deep Learning

Which of the following are activation functions?

Learn Before

Related

Learn After