Input: $$A^{[l-1]}$$
Output: $$A^{[l]}$$      Cache: $$Z^{[l]}, A^{[l-1]}$$
$$Z^{[l]} = W^{[l]} A^{[l-1]} + b^{[l]}$$
$$A^{[l]} = g^{[l]}(Z^{[l]})$$

Forward Propagation Formulation

Input:

$$dA^{[l]}$$

   For the last layer (L): 

$$dA^{[L]} = \sum_{i=1}^{m} -\frac{y^{[1]}}{a^{[1]}} + \frac{1 - y^{[1]}}{1 - a^{[1]}}$$

Output: 

$$dA^{[l-1]},   dW^{[l]},   db^{[l]}$$

$$dZ^{[l]} = dA^{[l]} g'^{[l]}(Z^{[l]})$$

$$dW^{[l]} = \frac{1}{m} dZ^{[l]} A^{[l - 1] T}$$

$$db^{[l]} = \frac{1}{m} \sum dZ^{[l]}$$

$$dA^{[l - 1]} = W^{[l]T} dZ^{[l]}$$ 

Backward Propagation Formulation

In the general case what is the dimension of $$W^{[l]}$$, the weight matrix associated with layer ll? (hint: $$n^{[l]}$$ is the number of inputs in l layer)

Dimension of weight matrix

$L$ = # of layers
$m$ = # of training datapoints
$$n^{[l]} = $$ # of units (neurons) in layer $l$
$$A^{[l]} = g^{[l]}(Z^{[l]}) = $$ activations (outputs) in layer $l$
$Z^{[l]}, A^{[l]}: (n^{[l]},m)$
$$X = A^{[0]}$$      $$\hat{Y} = A^{[L]}$$
$$W^{[l]} = $$ Weights for $$Z^{[l]}$$      $$W^{[l]}: (n^{[l]}, n^{[l - 1]})$$
$$b^{[l]} = $$ Biases for $$Z^{[l]}$$      $$b^{[l]}: (n^{[l]}, 1)$$

University of Michigan - Ann Arbor

Feedforward neural networks are called networks because they are typically represented by composing together many different functions. The model is associated with a directed acyclic graph describing how the functions are composed together.

For example, we might have three functions $f^{(1)} , f ^{(2)}$, and $f^{(3)}$ connected in a chain, to form $f (x) = f ^{(3)}(f^{(2)}(f^{(1)}(x)))$. These chain structures are the most commonly used structures of neural networks. 


Overall Structure of Deep Feedforward Networks

For a feedforward neural network, the depth of the network is the number of hidden layers plus one (as the output layer is also parameterized). The width of the network is the dimensionality of its hidden layers.

The main architectural considerations for designing a neural network are choosing the depth of the network and the width of each layer.


Depth and Width for Neural Networks

In the chain structure example $f (x) = f ^{(3)}(f^{(2)}(f^{(1)}(x)))$, $f^{(1)}$ is called
the first layer of the network, f^{(2)} is called the second layer, and so on. The final layer of a feedforward network is called the output layer. 

During neural network training, we drive $f(x)$ to match $f^∗(x)$. The training data provides us with noisy, approximate examples of $f^∗(x)$ evaluated at different training points. Each example x is accompanied by a label $y ≈ f^*(x)$.

The training examples specify directly that the output layer must produce a value that is close to $y$ at each point $x$. However, the behavior of the other layers is not directly specified by the training data. The training data do not say what each individual layer should do. Instead, the learning algorithm must decide how to use these layers to best implement an approximation of $f^*$. Because the training data does not show the desired output for each of these layers, they are called hidden layers.

Layers of a Feed Forward Neural Network

Formulation of a Feedforward Neural Networks

The universal approximation theorem (Horniket al., 1989; Cybenko, 1989) states that a feedforward network with a linear output layer and at least one hidden layer with any “squashing” activation function (such as the logistic sigmoid activation function) can approximate any Borel measurable function from one ﬁnite-dimensional space to another with any desired nonzero amount of error, provided that the network is given enough hidden units. The derivatives of the feedforward network can also approximate the derivatives of the function arbitrarily well.

It means that regardless of what function we are trying to learn, we know that a large feedforward network will be able to represent this function. 

Learn Before

Related

Learn After