Learn Before
Pre-training Objective for Language Models
The loss function initially defined for a single token sequence can be extended to an entire set of sequences, denoted as . In this context, the primary objective of the pre-training process is to discover the optimal set of model parameters that minimizes the total loss calculated across all sequences within .
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Pre-training Objective for Language Models
Example of a Token Sequence
Example of an Indexed Token Sequence
A language model is evaluated on a sequence of four tokens,
(x_0, x_1, x_2, x_3). The model's performance is measured by calculating a loss value at each step of the sequence generation. The individual losses are as follows: the loss for predicting tokenx_1is 1.2, the loss for predictingx_2is 0.5, and the loss for predictingx_3is 2.3. Based on this information, what is the total loss for the entire token sequence?Comparative Model Performance Analysis
A language model's performance is being evaluated on the token sequence
('The', 'cat', 'sat', 'on'). The total loss for this sequence is calculated by summing the individual losses from each predictive step. Which of the following sets of predictions contributes to this total loss calculation?Ground-Truth Distribution as a One-Hot Representation
Learn After
Probability Computation with Pre-trained Language Models
A language model is being trained on a large dataset of text. After an initial training iteration, the model's performance is measured on three distinct sequences from the dataset, yielding the following loss values:
- Sequence 1: Loss = 8.4
- Sequence 2: Loss = 2.1
- Sequence 3: Loss = 5.5
Based on the fundamental objective of this training process, which of the following statements most accurately describes the model's overall goal?
Evaluating Model Training Progress
From Single Sequence to Full Dataset
The primary objective of pre-training a language model on a dataset is to find a unique, optimal set of model parameters for each individual text sequence within that dataset.
Pre-training Objective Formula