Learn Before
Absence of a Universal Scaling Law
There is no single, universally applicable scaling law that can precisely describe the relationship between training factors and LLM performance in all situations. The complexity of the training dynamics means that a one-size-fits-all formula has not been established.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Absence of a Universal Scaling Law
A research team is developing a new language model. They train several versions of the model, each with a different number of parameters, while keeping the training dataset size fixed. They plot the final training loss for each model version against its parameter count. The resulting graph shows a consistent, downward-curving trend: as the number of parameters increases, the loss decreases, but the amount of improvement gets smaller with each increase. Based on this observation, what is the most
Optimizing LLM Training Budget
Comparing Single-Variable Scaling Functions
Learn After
Limitations of Monotonic Scaling Functions
Limitation of Test Loss in Predicting Downstream Performance
A research team develops a scaling function that accurately predicts their language model's performance on English text as they increase the model's parameter count. Confident in their findings, they use the same function to budget for a new, larger model intended for generating computer code. However, the final code-generation model performs significantly worse than the function predicted. Which statement best explains this outcome?
Evaluating a Compute Budgeting Strategy
A research lab has developed a scaling function that accurately predicts the performance of their specific 10-billion parameter language model on a large corpus of web text. This function can therefore be considered a reliable predictor for the performance of any other 10-billion parameter language model trained on a different large corpus of web text.