The ultimate target for optimization during neural network training is the objective function, denoted as $$J$$. This function combines the network's predictive error with a complexity penalty. For a given data example, it is the sum of the unregularized loss term $$L$$ and the $$\ell_2$$ regularization term $$s$$:

$$J = L + s$$

Minimizing this regularized objective function balances fitting the training data accurately while maintaining small parameter weights to improve overall generalization.

Regularized Loss Objective Function

To evaluate the performance of a neural network on an individual data point, a loss function $$l$$ is applied to its predictions. Given the network's final output layer variable $$\mathbf{o}$$ and the true target label $$y$$, the unregularized loss term $$L$$ for a single example is formulated as:

$$L = l(\mathbf{o}, y)$$

This scalar value quantifies the discrepancy between the model's output and the actual ground truth for that specific data example before any complexity penalties are added.

Claude

The final linear transformation in a single hidden layer neural network occurs at the output layer. Using the hidden activation vector $$\mathbf{h}$$ as input, and assuming the output layer uses a weight parameter matrix $$\mathbf{W}^{(2)} \in \mathbb{R}^{q 	imes h}$$ (without a bias term), the output layer variable $$\mathbf{o}$$ of length $$q$$ is calculated as:

$$\mathbf{o} = \mathbf{W}^{(2)} \mathbf{h}$$

Output Layer Variable Formula

Dive into Deep Learning

Single Example Loss Term Formula

To compute the gradient of the objective function $$J$$ with respect to the output layer variable $$\mathbf{o} \in \mathbb{R}^q$$, the chain rule is applied through the loss term $$L$$. Because the gradient of $$J$$ with respect to $$L$$ is $$1$$, the formula simplifies directly to the partial derivative of the loss with respect to the output:
$$ \frac{\partial J}{\partial \mathbf{o}} = \textrm{prod}\left(\frac{\partial J}{\partial L}, \frac{\partial L}{\partial \mathbf{o}}\right) = \frac{\partial L}{\partial \mathbf{o}} $$

Learn Before

Related

Learn After