Calculating Loss Reduction from Increased Dataset Size
A research team is training a large language model and finds that its loss (L) follows the empirical power law with respect to dataset size (D): If the team doubles the size of their training dataset, by what approximate percentage will the model's loss decrease? Provide your final answer and the key calculation step.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Combined Power Law for LLM Loss with Model and Dataset Size
Predicting LLM Performance Based on Dataset Size
A research team observes that their language model's loss (L) decreases as the training dataset size (D) increases, following the specific power law: where C is a large constant and the exponent α is a small positive number (e.g., 0.095). Based on this mathematical relationship, what is the most significant implication for the team as they consider scaling up their training data from an already very large starting point?
Calculating Loss Reduction from Increased Dataset Size
Power Law Fit for Test Loss vs. Model and Dataset Size