Learn Before
From Single Sequence to Full Dataset
A language model's performance can be measured by calculating a 'loss' value for a single sequence of text. Explain how this single-sequence measurement is used to define the main goal of the entire pre-training process, which involves a massive dataset containing millions of such sequences.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Comprehension in Revised Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Probability Computation with Pre-trained Language Models
A language model is being trained on a large dataset of text. After an initial training iteration, the model's performance is measured on three distinct sequences from the dataset, yielding the following loss values:
- Sequence 1: Loss = 8.4
- Sequence 2: Loss = 2.1
- Sequence 3: Loss = 5.5
Based on the fundamental objective of this training process, which of the following statements most accurately describes the model's overall goal?
Evaluating Model Training Progress
From Single Sequence to Full Dataset
The primary objective of pre-training a language model on a dataset is to find a unique, optimal set of model parameters for each individual text sequence within that dataset.
Pre-training Objective Formula