A binary step function is a threshold-based activation function. If the input value is above or below a certain threshold, the neuron is activated and sends exactly the same signal to the next layer.

Binary Step Function

A linear activation function takes the form: $y = cx$.

It takes the inputs, multiplied by the weights for each neuron, and creates an output signal proportional to the input. In one sense, a linear function is better than a step function because it allows multiple outputs, not just yes and no.


Linear Activation Function

Modern neural network models use non-linear activation functions. They allow the model to create complex mappings between the network’s inputs and outputs, which are essential for learning and modeling complex data, such as images, video, audio, and data sets which are non-linear or have high dimensionality.

There are 8 common non-linear activation functions:
 - Sigmoid / Logistic
 - TanH / Hyperbolic Tangent
 - ELU (Exponential Linear Unit)
 - ReLU (Rectified Linear Unit)
 - Leaky ReLU
 - Parametric ReLU
 - Softmax
 - Swish

Non-Linear Activation Functions

Sigmoid 
Pros: useful for binary classification, output layer 
Cons: shifted from center

Tanh 
Pros: has centering effect, almost always superior to sigmoid 
Cons: complex in comparison to ReLU 

ReLU
Pros: simpler, default
Cons: Derivative is 0 when z is negative --> leaky reLU

Pros and Cons of each Activation Function

Activation functions are a critical part of the design of a neural network- A hidden layer in a neural network is a layer that receives input from another layer (such as another hidden layer or an input layer) and provides output to another layer (such as another hidden layer or an output layer).

There are perhaps three activation functions you may want to consider for use in hidden layers; they are:

- Rectified Linear Activation (ReLU)
The rectified linear activation function calculated as max(0.0, x)
It is common because it is both simple to implement and effective at overcoming the limitations of other previously popular activation functions, such as Sigmoid and Tanh.


- Logistic (Sigmoid)
The sigmoid activation function is also called the logistic function, and calculated as 1.0 / (1.0 + e^-x)
It is used in the logistic regression classification algorithm.

- Hyperbolic Tangent (Tanh)
The function takes any real value as input and outputs values in the range -1 to 1. The larger the input (more positive), the closer the output value will be to 1.0, whereas the smaller the input (more negative), the closer the output will be to -1.0.
IT is calculated as (e^x – e^-x) / (e^x + e^-x)

It is common practive to choose activation function based on the type of model you work on.
Multilayer Perceptron (MLP): ReLU activation function.
Convolutional Neural Network (CNN): ReLU activation function.
Recurrent Neural Network: Tanh and/or Sigmoid activation function.



How to Choose an Activation Function for Deep Learning

Which of the following are activation functions?

There are 12 types of activation functions:

- Binary Step
- Linear
- Sigmoid
- Tanh
- ReLU
- Leaky ReLU
- Parameterised ReLU
- Exponential Linear Unit
- Swish
- Softmax

https://www.analyticsvidhya.com/blog/2020/01/fundamentals-deep-learning-activation-functions-when-to-use-them/

University of Michigan - Ann Arbor

In a neural network, numeric data points, called inputs, are fed into the neurons in the input layer. Each neuron has a weight, and multiplying the input number with the weight gives the output of the neuron, which is transferred to the next layer.
The activation function is a mathematical “gate” in between the input feeding the current neuron and its output going to the next layer. It can be as simple as a step function that turns the neuron output on and off, depending on a rule or threshold. Or it can be a transformation that maps the input signals into output signals that are needed for the neural network to function.
Activation functions are mathematical equations that determine the output of a neural network. The function is attached to each neuron in the network, and determines whether it should be activated (“fired”) or not, based on whether each neuron’s input is relevant for the model’s prediction. Activation functions also help normalize the output of each neuron to a range between 1 and 0 or between -1 and 1. When used in a multilayer neural network, activation functions can be different for different layers. 

Activation Functions in Neural Networks

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

Types of Activation Functions

In a neural network, numeric data points, called inputs, are fed into the neurons in the input layer. Each neuron has a weight, and multiplying the input number with the weight gives the output of the neuron, which is transferred to the next layer.

The activation function is a mathematical “gate” in between the input feeding the current neuron and its output going to the next layer. It can be as simple as a step function that turns the neuron output on and off, depending on a rule or threshold. Or it can be a transformation that maps the input signals into output signals that are needed for the neural network to function.

Role of the Activation Functions in Neural Networks

https://engmrk.com/activation-function-for-dnn/

How to Select an Activation Function

A neural network with multiple hidden layers is designed so that for every neuron, its output is simply the direct weighted sum of its inputs. No further mathematical transformation is applied to this sum before it is passed to the next layer. What is the most significant consequence of this design on the network's overall capability?

An engineer is training a deep neural network and observes that the raw output values from many neurons (the weighted sum of their inputs) are becoming extremely large, leading to an unstable training process where the model fails to learn effectively. To solve this, a specific type of mathematical function is typically applied to the raw output of each neuron before it is passed to the next layer.

Diagnosing Neural Network Instability

A data scientist is building a neural network with several hidden layers to classify images. They consider two approaches for the hidden layers. In Approach A, the output of each neuron is the direct weighted sum of its inputs. In Approach B, a non-linear mathematical function is applied to the weighted sum before passing the result to the next layer. Which approach is fundamentally more capable of learning the complex patterns in the images, and why? Explain the critical limitation of the less capable approach.

Learn Before

Related

Learn After