The data used to train the model are known as parallel texts, or bitexts.
bitexts = source + </s> + target
, where source is the text being translated, target is the translation output, and </s> is the end-of-sentence token.

Application of autoregressive generation given a prefix: Machine translation

The abstracted architecture of the twisted autoregressive generation is referred to as the encoder-decoder architecture.

It consists of three parts:
- encoder: accepts a sequence as its input and generates a corresponding sequence of contextualized representations (hidden states);
- context vector: is a function of the vector of contextualized representations generated from the encoder and conveys the essence of the input to the decoder;
- decoder: takes context vector as input and generates an arbitrary length sequence of hidden states, therefore obtains a corresponding sequence of output states.

The encoder and decoder networks are typically implemented with the same architecture, often using recurrent networks, but there are some other possibilities in each part.

Encoder-decoder networks

Instead of generating a sentence from scratch, the language model can complete a sequence given a specified prefix as following.

1. Pass a specified prefix to the language model using forward inference to produce a sequence of hidden states;
2. Apply autoregressive generation using the hidden state of the last word of the prefix as the starting point of generation;
3. The result of this process is a sequence of words that should be a reasonable completion given the prefix input.

University of Michigan - Ann Arbor

The technique (Shannon, 1951) generates a word in a sequence given the previous words, using cross-entropy as the loss function and perplexity as the evaluation method. The sequence ends once it reaches a predefined length. Each softmax layer is to decide the next word based on the current hidden state in the RNN component. The initial input token(s) would depend on the NLP task's nature (e.g. machine translation, question answering, summarization). 

RNN's Approach to Text Generation in NLP (Autoregressive Generation)

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Shannon, C. E. (1951). Prediction and entropy of printed English. Bell System Technical Journal, 30, 50–64.

Learn Before

Related

Learn After