The final linear transformation in a single hidden layer neural network occurs at the output layer. Using the hidden activation vector $$\mathbf{h}$$ as input, and assuming the output layer uses a weight parameter matrix $$\mathbf{W}^{(2)} \in \mathbb{R}^{q 	imes h}$$ (without a bias term), the output layer variable $$\mathbf{o}$$ of length $$q$$ is calculated as:

$$\mathbf{o} = \mathbf{W}^{(2)} \mathbf{h}$$

Output Layer Variable Formula

To continue backpropagation backwards towards the input layer, we calculate the gradient of the objective function $$J$$ with respect to the hidden layer output vector $$\mathbf{h} \in \mathbb{R}^h$$. By applying the chain rule through the output layer variable $$\mathbf{o}$$, we obtain:
$$ \frac{\partial J}{\partial \mathbf{h}} = \textrm{prod}\left(\frac{\partial J}{\partial \mathbf{o}}, \frac{\partial \mathbf{o}}{\partial \mathbf{h}}\right) = {\mathbf{W}^{(2)}}^\top \frac{\partial J}{\partial \mathbf{o}} $$
This operation successfully propagates the error gradient backward by multiplying it with the transpose of the output layer's weight matrix.

Gradient of Objective Function with Respect to Hidden Layer Output

Following the calculation of the linear intermediate variable $$\mathbf{z}$$, a neural network applies a non-linear activation function $$\phi$$ to produce the hidden layer's output. The hidden activation vector $$\mathbf{h}$$ of length $$h$$ is defined as:

$$\mathbf{h} = \phi(\mathbf{z})$$

This vector $$\mathbf{h}$$ serves as another intermediate variable in the network's forward pass, containing the activated representations that will be passed to the subsequent layer.

Claude

In a neural network with a single hidden layer, the computation begins by calculating an intermediate variable for the hidden layer. Assuming the input example is represented by a vector $$\mathbf{x} \in \mathbb{R}^d$$ and no bias term is included, the intermediate variable $$\mathbf{z} \in \mathbb{R}^h$$ is obtained by multiplying the hidden layer's weight parameter matrix $$\mathbf{W}^{(1)} \in \mathbb{R}^{h 	imes d}$$ by the input vector:

$$\mathbf{z} = \mathbf{W}^{(1)} \mathbf{x}$$

Intermediate Variable Formula

Dive into Deep Learning

Hidden Activation Vector Formula

Because a neural network's activation function $$\phi$$ applies elementwise, computing the gradient of the objective function $$J$$ with respect to the pre-activation intermediate variable $$\mathbf{z} \in \mathbb{R}^h$$ requires elementwise multiplication (denoted by $$\odot$$). Using the chain rule through the hidden layer output $$\mathbf{h}$$, the gradient is calculated as: $$ \frac{\partial J}{\partial \mathbf{z}} = 	extrm{prod}\left(\frac{\partial J}{\partial \mathbf{h}}, \frac{\partial \mathbf{h}}{\partial \mathbf{z}}ight) = \frac{\partial J}{\partial \mathbf{h}} \odot \phi'\left(\mathbf{z}ight) $$ where $$\phi'(\mathbf{z})$$ represents the local derivative of the activation function.

Learn Before

Related

Learn After