A research team is training a 10-billion parameter language model. After consuming 25% of their total compute budget, they observe that the model's performance improvement, when plotted against the compute used, is tracking perfectly along the curve predicted by established scaling laws. However, this predicted trajectory indicates that the model will fall short of its target performance goal by the time 100% of the budget is used. Based on the predictive utility of scaling laws, what is the mos
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Optimizing LLM Training with a Fixed Budget
A research team is training a 10-billion parameter language model. After consuming 25% of their total compute budget, they observe that the model's performance improvement, when plotted against the compute used, is tracking perfectly along the curve predicted by established scaling laws. However, this predicted trajectory indicates that the model will fall short of its target performance goal by the time 100% of the budget is used. Based on the predictive utility of scaling laws, what is the mos
Strategic Implications of Scaling Law Predictions in LLM Training