Self-attention layers' first approach

Transformers in contextual generation and summarization

Huggingface Model Summary

A Survey of Transformers (Lin et. al, 2021)

Overview of a Transformer

Model Usage of Transformers

Attention in vanilla Transformers

Transformer Variants (X-formers)

The Pre-training and Fine-tuning Paradigm

Architectural Categories of Pre-trained Transformers

Computational Cost of Self-Attention in Transformers

Quadratic Complexity's Impact on Transformer Inference Speed

Pre-Norm Architecture in Transformers

Critique of the Transformer Architecture's Core Limitation

A research team is building a model to summarize extremely long scientific papers. They are comparing two distinct architectural approaches:

• Approach 1: Processes the input text sequentially, token by token, updating an internal state that is passed from one step to the next.
• Approach 2: Processes all input tokens simultaneously, using a mechanism that directly relates every token to every other token in the input to determine context.

Which of the following statements best analyzes the primary trade-off between these two approaches for this specific task?

Architectural Design Choice for Machine Translation

Enablers of Universal Language Capabilities

Model Depth in Transformers

Generalization of the Language Modeling Concept

Transformer Block Sub-Layers

Standard Optimization Objective for Transformer Language Models

Comparison of Self-Supervised Pre-training and Self-Training

Architectural Categories of Pre-trained Transformers

Self-Supervised Classification Tasks for Encoder Training

Prefix Language Modeling (PrefixLM)

Mask-Predict Framework

Discriminative Training

Learning World Knowledge from Unlabeled Data

Emergent Linguistic Capabilities from Pre-training

Architectural Approaches to Self-Supervised Pre-training

Self-Supervised Pre-training of Encoders via Masked Language Modeling

Word Prediction as a Core Self-Supervised Task

Learning World Knowledge from Unlabeled Data via Self-Supervision

A research team has a massive collection of unlabeled historical texts. Their goal is to pre-train a language model that understands the specific vocabulary and sentence structures within these documents, but they have no budget for manual data annotation. Which of the following approaches is the most effective and feasible for their pre-training task?

Analysis of Supervision Signal Generation

A team is developing a pre-training strategy for a new language model using a large corpus of unlabeled text. Which of the following proposed tasks best exemplifies the principles of self-supervised learning?

Prevalence of Self-Supervised Pre-training in NLP