A data scientist is training a model to perform part-of-speech tagging, where the goal is to assign a grammatical label (e.g., noun, verb, adjective) to every word in an input sentence. They are considering two different methods for calculating the total loss for each training sentence:

Google

Given the predictions for each time step $\hat{y}^{(t)}$, together with the ground truth labels $y^{(t)}$, we can calculate the loss on step t
$$J^{(t)} = \mathcal{L} (\hat{y}^{(t)}, y^{(t)})$$
To get the overall loss, we only need to average these:
$$J = \frac{1}{T} \sum_{t=1}^T J^{(t)} $$

Loss Function for RNN

In an RNN, backpropagation needs to take into account that the model is carrying forward information from each neural layer to the next, and fine tune the weights that govern this “short-term memory”.

This is called Backpropagation Through Time (BPTT). BPTT uses the chain rule to go back from the latest time step to the previous step, and then to the next-previous, each time using gradient descent to discover the best weights for each neuron and for the hidden state function.

Backpropagation Through Time (BPTT)

A model designed to process sequential data is evaluated on a sequence of 4 time steps. The loss (error) is calculated independently at each time step, yielding the following values: [0.2, 0.5, 0.1, 0.4]. Based on the standard method for computing the total loss for the entire sequence, what is the final loss value?

Evaluating Loss Calculation Strategies

In a model that processes sequential data, why is the overall performance typically evaluated by averaging the error from each individual step in the sequence, rather than by only considering the error at the very last step?

Learn Before

Related