University of Michigan - Ann Arbor

When we use a feedforward neural network to accept an input $x$ from training set and produce an output $\hat{y}$, information ﬂows forward through the network. The input $x$ provides the initial information that then propagates up to the hidden units at each layer and ﬁnally produces $\hat{y}$. This is called forward propagation. 


Forward Propagation

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

In the chain structure of a feedforward network, Let's assume the input example is $x \in \mathbb{R}^d$, then the first layer is given by
$h^{(1)} = g^{(1)} (W^{(1)T}x + b^{(1)})$
the second layer is given by 
$h^{(2)} = g^{(2)} (W^{(2)T} h^{(1)}+ b^{(2)})$
and so on.
For the output layer,
$\hat{y} = g^{(n)}(W^{(n)T} h^{(n-1)}+ b^{(n)})$

where the matrix $W^{(k)}\in \mathbb{R}^{d\times h_k}$ is the weight parameter for layer $k$ and $g^{k}$ is the activation function for layer $k$. Whether transposing $W^{(k)}$ depends on the shape of the vector.

Connection between the Layers of Neural Network

Input: $$A^{[l-1]}$$
Output: $$A^{[l]}$$      Cache: $$Z^{[l]}, A^{[l-1]}$$
$$Z^{[l]} = W^{[l]} A^{[l-1]} + b^{[l]}$$
$$A^{[l]} = g^{[l]}(Z^{[l]})$$

Forward Propagation Formulation

True/False: During forward propagation, in the forward function for a layer ll you need to know what is the activation function in a layer (Sigmoid, tanh, ReLU, etc.). During back propagation, the corresponding backward function also needs to know what is the activation function for layer ll, since the gradient depends on it.

Which of these is a correct vectorized implementation of forward propagation for layer $\ell$, where 1≤$\ell$≤$\mathcal{L}$?

- Z$^{[\ell]}$ = W$^([\ell])$A$^([\ell])$ + b$^([\ell])$


Learn Before

Related