Comparison of Self-Supervised Pre-training and Self-Training

Architectural Categories of Pre-trained Transformers

Self-Supervised Classification Tasks for Encoder Training

Prefix Language Modeling (PrefixLM)

Mask-Predict Framework

Discriminative Training

Learning World Knowledge from Unlabeled Data

Emergent Linguistic Capabilities from Pre-training

Architectural Approaches to Self-Supervised Pre-training

Self-Supervised Pre-training of Encoders via Masked Language Modeling

Word Prediction as a Core Self-Supervised Task

Learning World Knowledge from Unlabeled Data via Self-Supervision

A research team has a massive collection of unlabeled historical texts. Their goal is to pre-train a language model that understands the specific vocabulary and sentence structures within these documents, but they have no budget for manual data annotation. Which of the following approaches is the most effective and feasible for their pre-training task?

Analysis of Supervision Signal Generation

A team is developing a pre-training strategy for a new language model using a large corpus of unlabeled text. Which of the following proposed tasks best exemplifies the principles of self-supervised learning?

Prevalence of Self-Supervised Pre-training in NLP