Google

Within Natural Language Processing, pre-trained models based on the Transformer are commonly categorized by their underlying architecture. These primary categories, which are targets for self-supervised pre-training approaches, include encoder-only, decoder-only, and encoder-decoder structures.

Architectural Categories of Pre-trained Transformers

*Encoder only* -> natural language understanding
- BERT
- RoBERTa (removes NSP task from BERT)

*Decoder only* -> language modeling
- GPT series

*Encoder-Decoder* -> equipped with the ability to perform both natural language understanding and generation
- BART
- T5

Examples of Pre-trained Transformers by Architecture

Imagine you are designing a system for two different natural language tasks.

*   **Task A:** Classifying the sentiment of a customer review (e.g., positive or negative). This requires a deep understanding of the entire review's context.
*   **Task B:** Translating a sentence from English to French. This requires understanding the source sentence and then generating a new sentence in the target language.

Compare and contrast the suitability of an "encoder-only" architecture versus an "encoder-decoder" architecture for these two tasks. In your analysis, explain how the flow of information in each architecture makes it either well-suited or poorly-suited for the specific demands of Task A and Task B.

Architectural Suitability for Language Processing Tasks

A research team is developing a system to automatically summarize long scientific articles. The system needs to first read and understand the entire source article and then generate a concise, new paragraph that captures the key findings. Which architectural category of pre-trained model is fundamentally best suited for this kind of sequence-to-sequence task?

Learn Before

Related