Google

Transformer-based models have become the mainstream approach for pre-trained language models (PLMs) due to their superior performance. These powerful models are pre-trained on extensive text corpora using various word prediction tasks, such as masked language modeling (MLM). Once pre-trained, they can be successfully adapted and applied to a wide range of downstream NLP tasks. These PLMs are often categorized into three main architectural types.

Representative Transformer-based PLMs


This type of LM destroys the input text (e.g. masking words in sentence and trying to reconstruct the original text). It aims to build bidirectional encoding representations of the entire sentences, so infrastructures often correspond to the encoder part of transformer, where all input can be accessed at each location. They have been fine-tuned successfully for downstream tasks. Examples include BERT, ROBERTA, ERNIE, and applications are suitable for NLU tasks like sentence classification and sequence labeling. 

Auto-Encoding (AE) Models


The main task of these models is to predict the next word based on previous words, and they are represented by the GPT model family. The AR infrastructures are composed of the Transformer’s decoder part, so a masking mechanism is used in the training phase, causing attention calculations to only see the content before a word. Applications are suitable for NLG tasks. 


Auto-Regressive (AR) Models 

Sequence-to-sequence (seq2seq) models, which utilize both an encoder and a decoder, are a standard framework for text generation tasks. This approach is suitable for applications like machine translation, summarization, question answering, and dialogue generation, where a source text is mapped to a target text. Models like T5 and mBART are prominent examples of pre-trained seq2seq models. This framework is versatile, allowing both Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks to be addressed and fine-tuned within the same architecture.

Seq2seq Models for Text Generation

An engineering team is tasked with creating a system to analyze customer reviews and automatically classify them as 'positive', 'negative', or 'neutral'. The most critical requirement is for the model to have a deep, holistic understanding of the entire review's context to make an accurate classification. Which of the following architectural approaches for a pre-trained model would be best suited for this task?

You are an NLP engineer selecting a pre-trained model architecture for three different projects. Match each project description to the most suitable underlying model training objective.

Analyze the following scenario and explain the fundamental mismatch between the chosen model's pre-training objective and the desired application.

Learn Before

Related