A research team is pre-training a new large language model using a massive text corpus and significant computational resources. They are debating whether to include an objective where the model must predict if two text segments are consecutive in the original source. Based on key findings from large-scale model pre-training experiments, what is the most well-supported conclusion the team should reach regarding this objective?

Optimizing Pre-training Objectives

For any large language model pre-training process, regardless of the scale of data and computation used, including an objective to predict if two sentences are sequential is essential for achieving optimal performance on downstream tasks because it is the primary mechanism for the model to learn inter-sentence coherence.