RoBERTa's Key Findings on Scaling

Impact of Removing NSP Loss in RoBERTa

General Direction for Pre-training: Scaling Simple Tasks

A research team is building a new language model for natural language understanding tasks. They have a fixed model architecture and a large computational budget. They are debating the most effective pre-training strategy. Based on the primary findings demonstrated by subsequent improvements on encoder-only models, which approach is most likely to yield the best performance?

Diagnosing Pre-training Issues in Large-Scale Models

Optimizing Pre-training Objectives for Large-Scale Models