In a neural network with a single hidden layer, the computation begins by calculating an intermediate variable for the hidden layer. Assuming the input example is represented by a vector $$\mathbf{x} \in \mathbb{R}^d$$ and no bias term is included, the intermediate variable $$\mathbf{z} \in \mathbb{R}^h$$ is obtained by multiplying the hidden layer's weight parameter matrix $$\mathbf{W}^{(1)} \in \mathbb{R}^{h 	imes d}$$ by the input vector:

$$\mathbf{z} = \mathbf{W}^{(1)} \mathbf{x}$$

Intermediate Variable Formula

As the name suggests, it simply means that the input data is pushed in the forward direction of the network. The data will never flow in the reverse direction during the output generation.

To help further this idea: consider using a feed-forward neural network.


University of Minnesota - Twin Cities

- Neural Networks
- Forward Propagation
- Backward Propagation
- Activation Functions

Foundational Concepts of Deep Learning

Breaking this down into to components:  

- What are Neurons? 
- How are they connected?


 - At first, we can think view a neural network as how our brains work like neurons signaling one another (Also the inspiration for its name). In a sense these models allow computer programs to recognize patterns and solve common problems.

- Neural Networks are comprised of node layers. It contains the input layer, one (or more) hidden layers, and an output layer.

- Nodes (neurons) are connected via arrows and each has an associated weight and threshold. If the node surpasses the threshold set, then that neuron is activated and the data is sent on the next layer. This process repeats until it goes to the output layer.


Neural Network

In simple terms, an activation function is added in the neural network in order to help the network learn complex patterns in the data. When comparing this to the brain a neuron sends signals to other neurons but what signals being sent depends on the conditions. The activation function is at the end taking in the data from the node and transforms it into an input that a new node (neuron) can take into the new layer.

- And of course, typically math is involved with coming up with these functions.

Learn Before

Related

Learn After