As an alternative to teacher forcing, sequence-to-sequence decoders can be trained by feeding the model's own predicted token from the previous time step as the current input. This approach aligns the training process more closely with how the model generates sequences autoregressively during inference.

Predicted Token Feedback in Decoder Training

Teacher forcing is a common training strategy for sequence models where the ground-truth token from a prior time step is used as input, rather than the model's own generated prediction. In an encoder–decoder architecture, this involves feeding the original target sequence directly into the decoder. Specifically, a special beginning-of-sequence token (e.g., `` `<bos>` ``) is prepended to the target sequence, excluding its final token. The decoder is then trained to predict the original target sequence shifted by one time step, ending with an end-of-sequence token (e.g., `` `<eos>` ``). This shifting method for self-supervised learning closely resembles standard language model training.

Claude

University of Michigan - Ann Arbor

In an RNN, backpropagation needs to take into account that the model is carrying forward information from each neural layer to the next, and fine tune the weights that govern this “short-term memory”.

This is called Backpropagation Through Time (BPTT). BPTT uses the chain rule to go back from the latest time step to the previous step, and then to the next-previous, each time using gradient descent to discover the best weights for each neuron and for the hidden state function.

Backpropagation Through Time (BPTT)

- For the decoder, autoregressive generation is used to output a sequence, an element at a time, until an end-of-sentence marker appears.
- Typically, use an LSTM or GRU-based RNN, where the context vector consists of the final hidden state of the encoder and is used to initialize the first hidden state of the decoder. 
- To avoid the fading influence of context vector during the decoding process, a solution is to add the context vector as a parameter to the computation of the current hidden state.
- In order to keep track of what has already been generated and what hasn’t, condition the output on three parts, the newly generated hidden state, the output generated at the previous state, and the encoder context.
- Beam search is used to optimize the output, preventing the unreliable result by independently choosing the argmax over a sequence.

Decoder

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Dive into Deep Learning

In a neural network with many time steps or layers, a gradient at the early layer is the product of all the terms from the later layers, which leads to an inherently unstable situation. Especially when the value of gradient has become so small, it no longer updates properly or is vanished eventually. Exploding gradient can be considered as the opposite of vanishing process. The updated weights using gradient descent become so large that they cause the whole network to become unstable, which leads to numerical overflow.

Vanishing/exploding gradient

A helpful website for understanding backpropagation in RNN:
http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/

Helpful Website for BPTT

Weight-tying is where you have a language model and use the same weight matrix for the input-to-embedding layer (the input embedding) and the hidden-to-softmax layer (the output embedding). The idea is that these two matrices contain essentially the same information, each having a row per word in the vocabulary.

In addition to providing improved perplexity results, this approach significantly reduces the number of parameters required for the model.

Weight typing

A computational graph visualizes the dependencies among model variables and parameters during the forward and backward computation of a recurrent neural network. For example, computing the hidden state at a specific time step depends on the current input, the hidden state from the previous time step, and the hidden layer weight parameters. To calculate gradients for training, the graph is traversed in the opposite direction of the arrows, applying the chain rule sequentially from the final output back to the initial inputs and weights.

Computational Graph of RNN Backpropagation Through Time

For a recurrent neural network, the gradient of the objective function $$L$$ with respect to the output layer weight parameter $$\mathbf{W}_\textrm{qh}$$ is calculated by summing the gradients across all time steps $$T$$. Because the objective function depends on $$\mathbf{W}_\textrm{qh}$$ via the sequence of outputs $$\mathbf{o}_1, \ldots, \mathbf{o}_T$$, we apply the chain rule to obtain:

$$ \frac{\partial L}{\partial \mathbf{W}_\textrm{qh}} = \sum_{t=1}^T \textrm{prod}\left(\frac{\partial L}{\partial \mathbf{o}_t}, \frac{\partial \mathbf{o}_t}{\partial \mathbf{W}_\textrm{qh}}\right) = \sum_{t=1}^T \frac{\partial L}{\partial \mathbf{o}_t} \mathbf{h}_t^\top $$

where $$\mathbf{h}_t^\top$$ is the transpose of the hidden state at time step $$t$$, and $$\partial L/\partial \mathbf{o}_t$$ is the gradient of the objective with respect to the model output at that time step.

Gradient of RNN Objective with Respect to Output Weights

Teacher Forcing

In the context of machine translation, the auto-regressive decoding process generates each token of the translated text (target language) by conditioning on both the complete source language sequence and the tokens that have already been generated in the target language.

Auto-regressive Decoding in Machine Translation

An autoregressive sequence generation model is tasked with producing an output. At each step, it calculates the probability for every possible next element and selects the single element with the highest probability before moving to the next step. What is the primary limitation of this step-by-step selection strategy?

An autoregressive decoder generates an output sequence one element at a time. At each step, the decision for the next element is typically conditioned on three key inputs: 1) the element generated in the previous step, 2) the decoder's current hidden state, and 3) the context derived from the entire input sequence. Explain the distinct purpose of each of these three inputs in this process.

Decoder Input Analysis

Given the case study, what is the most probable architectural reason for the decline in translation quality towards the end of the generated sentence?

Diagnosing Translation Degradation

Beam search is a sequence decoding strategy that strikes a compromise between the computational efficiency of greedy search and the optimality of exhaustive search. Instead of greedily picking the single most likely token or exploring all possible paths, beam search evaluates and retains a predetermined number of the most promising candidate sequences at each step of the generation process.

Learn Before

Related

Learn After