Probability Computation with Pre-trained Language Models

A language model is being trained on a large dataset of text. After an initial training iteration, the model's performance is measured on three distinct sequences from the dataset, yielding the following loss values:

• Sequence 1: Loss = 8.4
• Sequence 2: Loss = 2.1
• Sequence 3: Loss = 5.5

Based on the fundamental objective of this training process, which of the following statements most accurately describes the model's overall goal?

Evaluating Model Training Progress

From Single Sequence to Full Dataset

The primary objective of pre-training a language model on a dataset is to find a unique, optimal set of model parameters for each individual text sequence within that dataset.

Pre-training Objective Formula