To evaluate the performance of a neural network on an individual data point, a loss function $$l$$ is applied to its predictions. Given the network's final output layer variable $$\mathbf{o}$$ and the true target label $$y$$, the unregularized loss term $$L$$ for a single example is formulated as:

$$L = l(\mathbf{o}, y)$$

This scalar value quantifies the discrepancy between the model's output and the actual ground truth for that specific data example before any complexity penalties are added.

Single Example Loss Term Formula

To compute the gradient of the objective function $$J$$ with respect to the output layer variable $$\mathbf{o} \in \mathbb{R}^q$$, the chain rule is applied through the loss term $$L$$. Because the gradient of $$J$$ with respect to $$L$$ is $$1$$, the formula simplifies directly to the partial derivative of the loss with respect to the output:
$$ \frac{\partial J}{\partial \mathbf{o}} = \textrm{prod}\left(\frac{\partial J}{\partial L}, \frac{\partial L}{\partial \mathbf{o}}\right) = \frac{\partial L}{\partial \mathbf{o}} $$

Gradient of Objective Function with Respect to Output Layer Variable

The final linear transformation in a single hidden layer neural network occurs at the output layer. Using the hidden activation vector $$\mathbf{h}$$ as input, and assuming the output layer uses a weight parameter matrix $$\mathbf{W}^{(2)} \in \mathbb{R}^{q 	imes h}$$ (without a bias term), the output layer variable $$\mathbf{o}$$ of length $$q$$ is calculated as:

$$\mathbf{o} = \mathbf{W}^{(2)} \mathbf{h}$$

Claude

Following the calculation of the linear intermediate variable $$\mathbf{z}$$, a neural network applies a non-linear activation function $$\phi$$ to produce the hidden layer's output. The hidden activation vector $$\mathbf{h}$$ of length $$h$$ is defined as:

$$\mathbf{h} = \phi(\mathbf{z})$$

This vector $$\mathbf{h}$$ serves as another intermediate variable in the network's forward pass, containing the activated representations that will be passed to the subsequent layer.

Hidden Activation Vector Formula

Dive into Deep Learning

Output Layer Variable Formula

To continue backpropagation backwards towards the input layer, we calculate the gradient of the objective function $$J$$ with respect to the hidden layer output vector $$\mathbf{h} \in \mathbb{R}^h$$. By applying the chain rule through the output layer variable $$\mathbf{o}$$, we obtain:
$$ \frac{\partial J}{\partial \mathbf{h}} = \textrm{prod}\left(\frac{\partial J}{\partial \mathbf{o}}, \frac{\partial \mathbf{o}}{\partial \mathbf{h}}\right) = {\mathbf{W}^{(2)}}^\top \frac{\partial J}{\partial \mathbf{o}} $$
This operation successfully propagates the error gradient backward by multiplying it with the transpose of the output layer's weight matrix.

Learn Before

Related

Learn After