In a neural network, numeric data points, called inputs, are fed into the neurons in the input layer. Each neuron has a weight, and multiplying the input number with the weight gives the output of the neuron, which is transferred to the next layer.
The activation function is a mathematical “gate” in between the input feeding the current neuron and its output going to the next layer. It can be as simple as a step function that turns the neuron output on and off, depending on a rule or threshold. Or it can be a transformation that maps the input signals into output signals that are needed for the neural network to function.
Activation functions are mathematical equations that determine the output of a neural network. The function is attached to each neuron in the network, and determines whether it should be activated (“fired”) or not, based on whether each neuron’s input is relevant for the model’s prediction. Activation functions also help normalize the output of each neuron to a range between 1 and 0 or between -1 and 1. When used in a multilayer neural network, activation functions can be different for different layers. 

Activation Functions in Neural Networks

What does a neuron compute?

A neural network maps an input vector $$X$$ to an output vector $$Y$$ by stacking layers of nodes, each functioning as a logistic-regression unit: a weighted sum of inputs passed through an activation function. The hidden layers contain neurons, the connections between them, and the activation functions applied at each layer to transform the input into the output. The accompanying figure shows the logistic-regression representation of two of the network's four neurons.

University of Michigan - Ann Arbor

Google

An Artificial Neural Network (ANN) is a computational model inspired by the structure of biological neural networks in the brain, made up of interconnected neuron-like units organized in layers. Rather than following task-specific rules, an ANN learns to recognize patterns in data from examples, adjusting the connections between its units during training.

Artificial Neural Networks

Given X, we want $\hat{y}=P(y = 1|X)$   $0 \leq \hat{y} \leq 1$

Parameters: $w \in R^{n_{x}}, b \in R$

Output (Activation): $\hat{y}^{(i)} = a^{(i)} = \sigma (w^{T}x^{(i)} + b)$

Given $[(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), ..., (x^{(m)}, y^{(m)})]$, we want $\hat{y}^{i} \approx y^{i}$

$\sigma(z^{(i)}) = \frac{1}{1 + e^{-z^{(i)}}}$

If z is large positive, $\sigma(z^{(i)}) \approx \frac{1}{1 + 0} = 1$

If z is large negative, $\sigma(z^{(i)}) \approx \frac{1}{1 + Big Num} \approx 0$

Logistic Regression Formulation

Deep learning is a particular kind of machine learning that achieves great power and flexibility by representing the world as a nested hierarchy of concepts, with each concept defined in relation to simpler concepts, and more abstract representationss computed in terms of less abstract ones.


Deep Learning (in Machine learning)

Artificial Neural Networks Formulation

The goal of a neural network model is to approximate some function $$f^*$$. For example, for a classifier, $$y = f^*(x)$$ maps an input $$x$$ to a category $$y$$. A neural network model defines a mapping $$y = f(x; \theta)$$ and learns the values for the parameter $$\theta$$ that result in the best function approximation.

Artificial Neural Networks Objective

The McCulloch-Pitts model is an early, foundational computational model of artificial neurons developed by cyberneticists and neurophysiologists Warren McCulloch and Walter Pitts. This model predates modern deep learning but provided a natural starting point by abstracting the structure of biological neural systems into interconnected mathematical units.

McCulloch-Pitts Artificial Neuron Model

While early artificial neural networks drew heavily from biological nervous systems, modern deep learning is a broadly interdisciplinary field. Just as aeronautics innovation eventually moved beyond the direct study of birds, contemporary deep learning draws inspiration in equal or greater measure from mathematics, linguistics, psychology, statistics, and computer science, rather than relying exclusively on neuroscience.

Interdisciplinary Inspiration of Deep Learning

What are the parameters of logistic regression?

 $L(\hat{y}, y) = -(ylog(\hat{y}) + (1 - y)log(1 - \hat{y}))$
- If y = 1: $L(\hat{y}, y) = -log(\hat{y}) \Rightarrow log(\hat{y}) \approx 0 \rightarrow \hat{y}  \approx 1$
- If y = 0: $L(\hat{y}, y) = -log(1 - \hat{y}) \Rightarrow log(1 - \hat{y}) \approx 0 \rightarrow \hat{y} \approx 0$

Logistic Regression Loss Function

To train the parameters W and B of the logistic regression model, you need to define a cost function.

$$J(w, b) = \frac{1}{m} \sum_{i=1}^{m} L(\hat{y}^{(i)}, y^{(i)})$$

$$=-\frac{1}{m} \sum_{i=1}^{m} [y^{(i)}log(\hat{y}^{(i)}) + (1 - y^{(i)})log(1 - \hat{y}^{(i)})]$$

This loss function is Convex.

Learn Before

Related

Learn After