Google

Self-supervised pre-training can be examined through the lens of different neural network designs, specifically decoder-only, encoder-only, and encoder-decoder architectures. The focus of such analysis is often narrowed to the Transformer architecture, as it underpins the vast majority of modern pre-trained models in the field of Natural Language Processing.

Architectural Approaches to Self-Supervised Pre-training

A research team is building a model designed specifically for summarizing long scientific articles into a few concise paragraphs. The model must be able to process the entire source article to understand its full context before generating the summary. Given this requirement for a sequence-to-sequence task, which architectural approach would be the most effective choice for the model's pre-training and fine-tuning?

Match each architectural approach for self-supervised pre-training with the category of tasks it is primarily designed to handle.

Analyze the primary limitation of using a pre-trained model with an encoder-only architecture for the task described in the case study below, and explain your reasoning.

Learn Before

Related