Classical Seq2seq Models

It means mapping one sequence to another sequence.

University of Michigan - Ann Arbor

1. autoregressive-models
2. autoencoding-models
3. seq-to-seq-models
4. multimodal-models
5. retrieval-based-models
6. masked Language models

Types of Pretrained Language Model

https://huggingface.co/docs/transformers/model_summary

Huggingface Model Summary

Auto-regressive model is often formatted as:
$y(t) = \sum_{i=1}^n a_i y(t-i) + e(t)$
It uses linear combination of previous time-related variables to predict the variable in the time t.

Auto-regressive Model in NLP

It maps the input into a hidden state and try to reconstruct the input to get a better hidden state for representation learning.

Autoencoding Model

Seq2Seq Model

In the pre-training phase, an encoder model is trained using a self-supervision objective like Masked Language Modeling. The process begins by converting a corrupted input sequence, where some tokens are masked, into a sequence of embeddings. This embedding sequence is then fed into the encoder, which generates contextual vector representations for all input tokens. Finally, these representations are passed to an output layer, such as a Softmax model, which is trained to reconstruct the original masked tokens.

Learn Before

Related

Learn After