Learn Before
From Theory to Practice: Expected vs. Empirical Loss
When training a reward model, the theoretical loss is defined as an expectation over the entire data distribution. In practice, this is replaced by a summation over a collected dataset. Explain the primary reason for this substitution and describe the key assumption about the collected dataset that is necessary for this approximation to be considered valid.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Comprehension in Revised Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A team is training a model to predict user preferences between two generated text responses. The training objective is to minimize the average loss calculated over a collected dataset of preferences. However, the data collection was flawed, resulting in a dataset that primarily contains preferences from a very specific, non-representative group of users. What is the most significant risk of using the average loss on this particular dataset as the primary metric for training the model?
From Theory to Practice: Expected vs. Empirical Loss
If a dataset used for training a preference model is extremely large, the average loss calculated over this dataset is guaranteed to be a highly accurate approximation of the theoretical loss over the entire data distribution, even if the data was collected from a narrow, specific user group.