If only linear activation functions are used in a neural network then the network is no more expressive than standard logistic regression without any hidden layers, because then all that is being done is linear transformations of the input. Thus, the network can not be more expressive than linear regression. It's by adding nonlinearity that we are able to approximate other kinds of more complex functions.
If $\sigma(z^{[1]}) = z^{[1]}$:
$$a^{[1]} = z^{[1]} = W^{[1]} x + b^{[1]}$$
$$a^{[2]} = z^{[2]} = W^{[2]} a^{[1]} + b^{[2]}$$
$$\Rightarrow a^{[2]} = W^{[2]} (W^{[1]} x + b^{[1]}) + b^{[2]}$$
$$= W^{[2]} W^{[1]} x + W^{[2]} b^{[1]} + b^{[2]} = W' x + b'$$

Why use a non-Linear Activation function

Non-linear functions address the problems of a linear activation function:

 - They allow backpropagation because they have a derivative function which is related to the inputs.
 - They allow “stacking” of multiple layers of neurons to create a deep neural network. Multiple hidden layers of neurons are needed to learn complex data sets with high levels of accuracy.

University of Michigan - Ann Arbor

Modern neural network models use non-linear activation functions. They allow the model to create complex mappings between the network’s inputs and outputs, which are essential for learning and modeling complex data, such as images, video, audio, and data sets which are non-linear or have high dimensionality.

There are 8 common non-linear activation functions:
 - Sigmoid / Logistic
 - TanH / Hyperbolic Tangent
 - ELU (Exponential Linear Unit)
 - ReLU (Rectified Linear Unit)
 - Leaky ReLU
 - Parametric ReLU
 - Softmax
 - Swish

Non-Linear Activation Functions

A linear activation function takes the form: $y = cx$.

It takes the inputs, multiplied by the weights for each neuron, and creates an output signal proportional to the input. In one sense, a linear function is better than a step function because it allows multiple outputs, not just yes and no.


Linear Activation Function

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

Linear vs. Non-Linear Activation Functions

$S(x) = \frac{1}{1 +e^{-x}} = \frac{e^x}{e^x + 1}$ 


Sigmoid/Logistic Function

$S(x)  = tanh(x) = \frac{e^x-e^{-x}}{e^x + e^{-x}}$


TanH/Hyperbolic Tangent Function

The Swish function, introduced by Ramachandran et al. in 2017, is a mathematical function defined as the product of its input and the sigmoid function applied to a scaled version of the input. For a scalar input $x$, it is expressed as:

$ \operatorname{swish}(x) = x \cdot \operatorname{sigmoid}(\beta x) = \frac{x}{1+e^{-\beta x}}$

where $\beta$ is a constant or a trainable parameter. When applied element-wise to a vector $\mathbf{h}$ in neural networks, the formula is written as:

$\sigma_{\text{swish}}(\mathbf{h}) = \mathbf{h} \odot \text{Sigmoid}(\beta \mathbf{h})$

where $\odot$ denotes the element-wise product.

Swish Function

The Rectified Linear Unit (ReLU) is a common choice for the activation function $$\sigma(\cdot)$$ within the hidden layers of neural networks. It is defined to output the positive portion of its argument. When applied to an input vector $$\mathbf{h}$$, the ReLU function is given by the formula: $$\sigma_{\mathrm{relu}}(\mathbf{h}) = \mathrm{max}(0, \mathbf{h})$$.

ReLU (Rectified Linear Unit)

 If x > 0:
 $f(x)=x$
 If x $<=$  1:
$f(x) = α(e^x - 1)$

ELU (Exponential Linear Unit)

Which activation function is represented by each of these plots?

Which of the following introduces nonlinearity into neural networks?

To convert raw, unnormalized outputs $$\mathbf{o}$$ into valid probabilities, the softmax function applies an exponential function to each component and then normalizes them by their sum. The exponentiation ensures that all probabilities are non-negative, while the division ensures that they sum to $$1$$. Mathematically, the predicted probability distribution $$\hat{\mathbf{y}}$$ is defined as:

$$\hat{\mathbf{y}} = \mathrm{softmax}(\mathbf{o}) \quad 	extrm{where}\quad \hat{y}_i = \frac{\exp(o_i)}{\sum_j \exp(o_j)}$$

This guarantees that $$0 \le \hat{y}_i \le 1$$ and $$\sum_j \hat{y}_j = 1$$. Unlike other normalizations or the probit model, the softmax function preserves order and leads to a more well-behaved optimization problem.

Softmax Function

A linear activation function has two major problems:

1. Not possible to use backpropagation  (gradient descent) to train the model—the derivative of the function is a constant, and has no relation to the input, X. So it’s not possible to go back and understand which weights in the input neurons can provide a better prediction.

2. All layers of the neural network collapse into one—with linear activation functions, no matter how many layers in the neural network, the last layer will be a linear function of the first layer (because a linear combination of linear functions is still a linear function). So a linear activation function turns the neural network into just one layer.

A neural network with a linear activation function is simply a linear regression model. It has limited power and ability to handle complexity varying parameters of input data.

Learn Before

Related

Learn After