In an RNN, backpropagation needs to take into account that the model is carrying forward information from each neural layer to the next, and fine tune the weights that govern this “short-term memory”.

This is called Backpropagation Through Time (BPTT). BPTT uses the chain rule to go back from the latest time step to the previous step, and then to the next-previous, each time using gradient descent to discover the best weights for each neuron and for the hidden state function.

Backpropagation Through Time (BPTT)

A model designed to process sequential data is evaluated on a sequence of 4 time steps. The loss (error) is calculated independently at each time step, yielding the following values: [0.2, 0.5, 0.1, 0.4]. Based on the standard method for computing the total loss for the entire sequence, what is the final loss value?

A data scientist is training a model to perform part-of-speech tagging, where the goal is to assign a grammatical label (e.g., noun, verb, adjective) to every word in an input sentence. They are considering two different methods for calculating the total loss for each training sentence:

Evaluating Loss Calculation Strategies

In a model that processes sequential data, why is the overall performance typically evaluated by averaging the error from each individual step in the sequence, rather than by only considering the error at the very last step?

Rationale for Averaging Time-Step Losses

Given the predictions for each time step $\hat{y}^{(t)}$, together with the ground truth labels $y^{(t)}$, we can calculate the loss on step t
$$J^{(t)} = \mathcal{L} (\hat{y}^{(t)}, y^{(t)})$$
To get the overall loss, we only need to average these:
$$J = \frac{1}{T} \sum_{t=1}^T J^{(t)} $$

University of Michigan - Ann Arbor

Google

Recurrent neural networks, or RNNs, are a family of neural networks for processing sequential data. A recurrent neural network is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time add additional layers of comprehension on top of the previous inputs. 


Recurrent Neural Network (RNN)

A sequence-level loss is a type of loss function that computes the total error over an entire data sequence. Instead of evaluating each element of the sequence independently, it aggregates the individual losses from each step—often through summation or averaging—to provide a single measure of the model's performance on the whole sequence.

Sequence-Level Loss

https://www.coursera.org/learn/nlp-sequence-models

Sequence Models

Reference of Foundations of Large Language Models Course

RNNs are currently being applied to several applications, including:

 - Temporal analysis—time-series anomaly detection and time-series prediction
 - Computer vision—Image description, video tagging and video analysis
 - NLP—Sentiment analysis, speech recognition, language modeling, machine translation and text generation

Applications of RNN

The simplest form of RNN (“vanilla” RNN) is similar to a regular neural network, only it contains a loop that allows the model to carry forward results from previous neuron layers.

RNN Basic Structure

RNNs come in many variants, including:
 - Bidirectional RNNs
 - Memory Units (LSTMs) RNNs
 - Fully recurrent 
 - Elman/Jordan networks, or Simple Recurrent Networks (SRN)
 - Recursive neural network
 - Hopfield
 - Echo State Network (ESN)
 - Stacked RNNs
 - Hierarchical
 - Neural Turing machines (NTMs)
 - Differentiable Neural Computer (DNC) 
 - Recurrent Multi-Layer Perceptron (RMLP)
 - Independent RNN (IndRNN)
 - Neural history compressor 
 - Second order RNNs 
 - Gated recurrent unit (GRU)
 - Continuous-Time Recurrent Neural Network (CTRNN)
 - Multiple Timescales Recurrent Neural Network (MTRNN) 
 - Neural Network Pushdown Automata (NNPDA)

RNN Extensions and Types

Loss Function for RNN

Both RNNs and HMMs make use of dynamic state. This means that these models depend both on the input system and current state. Additionally, RNNs have distributed state acquiring them with larger memory and computational capacity(which isn't true about HMMs). 

RNNs(Recurrent Neural Networks) vs HMMs (Hidden Markov Models)

Unlike feedforward neural networks:

1. Make use of sequence of inputs by using their memory
2. Cost function is applied at each time step unlike feedforward neural networks were cost is applied only to the final output. 

RNNs vs Feedforward Neural Networks

CNNs and RNNs are not mutually exclusive, as both can perform classification of image and text inputs, creating an opportunity to combine the two network types for increased effectiveness. This is especially true if the input to be classified is visually complex with added temporal characteristics that a CNN alone would be unable to process.

Typically, when these two network types are combined, sometimes referred to as a CRNN, inputs are first processed by CNN layers whose outputs are then fed to RNN layers. CNN Long Short-Term Memory (LSTM) architectures are particularly promising, as they facilitate analysis of inputs over longer periods than could be achieved with lower-level RNN architecture types.

Currently, these hybrid architectures are being explored for use in applications like video scene labeling, emotion detection or gesture recognition, video identification or gait recognition, and DNA sequence prediction.

Hybrid of Convolutional and Recurrent Neural Network

Why is an RNN (Recurrent Neural Network) used for machine translation, say translating English to French? (Check all that apply.)

In an RNN model, predictions are only based on knowledge/features/inputs from previous nodes. However, information from the rest of the sentence might be useful in the previous step as well.




Example: In the following examples, name entity recognition of Teddy can be different based on information that come after 'Teddy".




He said, "Teddy Roosevelt was a great president"
He said, "Teddy bear are on sale!"

RNN Problem

If we consider Tx the length of input and Ty the length of output:

1- Many-to-many ( Tx=Ty)

2- Many-to_many (Tx!=Ty)

3- Many-to-one

4-One_to_Many


Different types of RNN (in terms of input/output)

Gradients propagated many times tend to vanish or explode. Recurrent networks involve the composition of the same function many times which can result in nonlinear behavior. The figure shows the function after each time the transition function has been composed.



Long Term Dependencies Problem

"Conditioning" in the context of sequence to sequence learning in RNNs is the process of computing the probability of obtaining the output sequence conditioned on the input sequence, or p(y|x). The network is used to model this conditional probability mapping

We can extend a model to represent a distribution P(y|x) by using 3 common methods:

* As an extra input at each time step, or
* As the initial state h(0), or
* Both


Modeling Sequences Conditioned on Context with RNNs

As mentioned in the introduction to long-term dependencies, there are 3 ways to address long-term dependencies, and 1 of them is "Leakage units and other multi-timescale strategies", which will be introduced in this unit

This idea has led to numerous approaches, I will introduce three approaches here.

1 - Adding Skip Connections through Time

Adding direct connections from variables in the distant past to current variables is one way to get coarse time scales.
The gradient may vanish or exponentially about the time step. The algorithm introduces circular connections with d-delay to alleviate this problem. The rate of exponential decrease of the derivative is now related to τ/d correlation rather than τ. Since there is both a delay and a single-step connection, the gradient may still explode exponentially into t. This allows the learning algorithm to capture longer dependencies.

2- Leaky Units and a Spectrum of Diﬀerent Time Scales

The idea is similar to the sliding average model and shadow variables in TensorFlow.
The sliding average actually contains information over a long period of time.
In the long-term dependence problem, we can use the sliding average instead of the real parameters, making more long-term information to be considered when the parameters are iterated.

3- Removing connection

Unlike skiping connections, which add edges, the unit can self-select favorable dependencies. Whereas a removing connection directly removes the connection of length 1, forcing the connection to be longer.

Leaky Units and Other Strategies for Multiple Time Scales

A Convolutional Recurrent Neural Network (CRNN) is a combination of a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN). 
-CNN are specialized networks for processing data that use a specialized, linear mathematical operation, also known as a convolution. 
-RNN is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time adds additional layers of comprehension on top of the previous inputs. 
-The goal of a CRNN is to take advantage of CNNs for local feature extraction and RNNs for temporal summarization of the extracted features.


Convolutional Recurrent Neural Network (CRNN) 

For the parameters of all hidden states in a recurrent neural network, a pooling function can be applied to take their element-wise mean:

$\mathbf{h}_{\text {mean }}=\frac{1}{n} \sum_{i=1}^{n} \mathbf{h}_{i}$

Element-wise max values is also an option for the pooling function.

Pooling Layer in RNN

One reason for the inability of RNNs to carry forward critical information is that the hidden layers are being asked to perform two tasks simultaneously: 
- Provide information useful for the current decision
- Updating and carrying forward information required for future decisions

Inability of RNNs to Carry Forward Critical Information

Stacked RNNs consist of multiple networks where the output of one layer serves as the input to a subsequent layer, as shown in Fig. 9.10.

Stacked RNNs generally outperform single-layer networks. One reason for this
success seems to be that the network induces representations at differing levels of abstraction across layers. The optimal number of stacked RNNs is specific to each application and to each training set. However, as the number of stacks is increased the training costs rise quickly.

Stacked RNNs

A bidirectional RNN combines two independent bidirectional RNN
independent RNNs, one where the input is processed from the start to the end, and the other from the end to the start. We then concatenate the two representations computed by the networks into a single vector that captures both the left and right contexts of an input at each point in time. 

Fig. 9.11 illustrates such a bidirectional network that concatenates the outputs of the forward and backward pass. Other simple ways to combine the forward and backward contexts include element-wise addition or multiplication. The output at each step in time thus captures information to the left and to the right of the current input. In sequence labeling applications, these concatenated outputs can serve as the basis for a local labeling decision.

Bidirectional RNNs

When evaluating a model on a specific training instance, the loss function is calculated solely over the target sub-sequence, $$\mathbf{y}_{\mathrm{sample}}$$, instead of the full sequence. For a model defined by parameters $$\hat{\theta}^+$$, this loss is expressed as the negative log-likelihood of the probability of generating the output sub-sequence, given the input sub-sequence $$\mathbf{x}_{\mathrm{sample}}$$. The formula is:

$$\mathcal{L}_{\hat{\theta}^+}(\mathrm{sample}) = -\log \mathrm{Pr}_{\hat{\theta}^+}(\mathbf{y}_{\mathrm{sample}}|\mathbf{x}_{\mathrm{sample}})$$

Sample-wise Negative Log-Likelihood Loss for a Sub-sequence

A frequently used loss function in knowledge distillation is the sequence-level loss, which often takes the form of cross-entropy. This loss measures the dissimilarity between the teacher model's output distribution, $\text{Pr}^t(\mathbf{y}|\mathbf{c}, \mathbf{z})$, and the student model's distribution, $\text{Pr}_{\theta}^s(\mathbf{y}|\mathbf{c}', \mathbf{z})$. The total loss is calculated by summing the log probability of the student's predictions over all possible output sequences $\mathbf{y}$, weighted by the teacher's probability for each sequence. The formula is expressed as: $$ \text{Loss} = \sum_{\mathbf{y}} \text{Pr}^t(\mathbf{y}|\mathbf{c}, \mathbf{z}) \log \text{Pr}_{\theta}^s(\mathbf{y}|\mathbf{c}', \mathbf{z}) $$

Cross-Entropy Loss for Knowledge Distillation

A language model is being trained to generate the four-word sentence 'The quick brown fox'. The model generates one word at a time, and the error (loss) is calculated at each step:

- Loss for 'The' = 0.1
- Loss for 'quick' = 0.3
- Loss for 'brown' = 0.2
- Loss for 'fox' = 0.4

To update the model's parameters, the training process computes a single, overall loss value for the entire sentence. Which statement best analyzes this method of calculating the overall loss?

The total loss for a given sequence of $$m$$ tokens $$(x_0, \dots, x_m)$$ is computed by summing the individual losses over each position from $$i=0$$ to $$m-1$$. At each position $$i$$, a loss function $$\mathcal{L}$$ measures the discrepancy between the model's predicted probability distribution for the next token ($$\mathbf{p}_{i+1}^{\theta}$$) and the actual ground-truth distribution ($$\mathbf{p}_{i+1}^{\mathrm{gold}}$$). This is expressed generally as:

$$ \mathrm{Loss}_{\theta}(x_0, \dots, x_m) = \sum_{i=0}^{m-1} \mathcal{L}(\mathbf{p}_{i+1}^{\theta}, \mathbf{p}_{i+1}^{\mathrm{gold}}) $$

In natural language processing, this loss function $$\mathcal{L}$$ is typically the log-scale cross-entropy loss, leading to the specific formula:

$$ \mathrm{Loss}_{\theta}(x_0, \dots, x_m) = \sum_{i=0}^{m-1} \mathrm{LogCrossEntropy}(\mathbf{p}_{i+1}^{\theta}, \mathbf{p}_{i+1}^{\mathrm{gold}}) $$

Total Loss Calculation for a Token Sequence

A model is tasked with predicting a sequence of four numbers. For a specific training example, the model's error (loss) for each number in the sequence is calculated as follows: [0.8, 0.2, 0.5, 0.5]. If the overall performance on this sequence is measured by taking the *average* of the individual errors, what is the final sequence-level loss for this example?

Calculating Average Sequence-Level Loss

A team is training a model to translate sentences. They are debating two methods for calculating the error and updating the model for each training sentence:

**Method 1:** For a given sentence, the model generates the full translation. The errors for each individual word in the generated translation are then summed up. This single, total error value is used to adjust the model's parameters.

**Method 2:** For a given sentence, the model generates the first word of the translation. The error for this single word is calculated, and the model's parameters are immediately adjusted. Then, it generates the second word, calculates the error, and adjusts the parameters again. This process repeats for every word in the translation.

Analyze these two methods. Which method is a more appropriate application of calculating error for a complete data sequence, and why is it generally more effective for training models on sequential tasks like translation?

Learn Before

Related

Learn After