1Cademy - Connection between the Layers of Neural Network

Learn Before

Forward Propagation
Layers of a Feed Forward Neural Network

Concept

Connection between the Layers of Neural Network

In the chain structure of a feedforward network, Let's assume the input example is $x \in \mathbb{R}^d$ , then the first layer is given by $h^{(1)} = g^{(1)} (W^{(1)T}x + b^{(1)})$ the second layer is given by $h^{(2)} = g^{(2)} (W^{(2)T} h^{(1)}+ b^{(2)})$ and so on. For the output layer, $\hat{y} = g^{(n)}(W^{(n)T} h^{(n-1)}+ b^{(n)})$

where the matrix $W^{(k)}\in \mathbb{R}^{d\times h_k}$ is the weight parameter for layer $k$ and $g^{k}$ is the activation function for layer $k$ . Whether transposing $W^{(k)}$ depends on the shape of the vector.