Google

- For the decoder, autoregressive generation is used to output a sequence, an element at a time, until an end-of-sentence marker appears.
- Typically, use an LSTM or GRU-based RNN, where the context vector consists of the final hidden state of the encoder and is used to initialize the first hidden state of the decoder. 
- To avoid the fading influence of context vector during the decoding process, a solution is to add the context vector as a parameter to the computation of the current hidden state.
- In order to keep track of what has already been generated and what hasn’t, condition the output on three parts, the newly generated hidden state, the output generated at the previous state, and the encoder context.
- Beam search is used to optimize the output, preventing the unreliable result by independently choosing the argmax over a sequence.

Decoder

Beam search, also referred to as beam decoding, is an algorithm that extends the greedy search approach by exploring multiple potential paths simultaneously. Rather than committing to the single most probable token at each generation step, beam search retains a predetermined number of the best candidate sequences, a parameter known as the beam width. This strategy helps mitigate the problem of high-quality sequences being discarded too early in the generation process.

Beam search

In the context of machine translation, the auto-regressive decoding process generates each token of the translated text (target language) by conditioning on both the complete source language sequence and the tokens that have already been generated in the target language.

Auto-regressive Decoding in Machine Translation

An autoregressive sequence generation model is tasked with producing an output. At each step, it calculates the probability for every possible next element and selects the single element with the highest probability before moving to the next step. What is the primary limitation of this step-by-step selection strategy?

An autoregressive decoder generates an output sequence one element at a time. At each step, the decision for the next element is typically conditioned on three key inputs: 1) the element generated in the previous step, 2) the decoder's current hidden state, and 3) the context derived from the entire input sequence. Explain the distinct purpose of each of these three inputs in this process.

Decoder Input Analysis

Given the case study, what is the most probable architectural reason for the decline in translation quality towards the end of the generated sentence?

Learn Before

Related