Learn Before
Example of Emergent Abilities Study (Wei et al., 2022b)
A notable study by Wei et al. (2022b) investigated the scaling properties of Large Language Models by varying model size and computational resources. The research demonstrated that certain advanced abilities only manifest once the model's scale reaches a particular threshold, providing concrete evidence for the concept of emergent abilities.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Related
Motivation for Continued Scaling of LLMs
Example of Emergent Abilities Study (Wei et al., 2022b)
A research lab trains a series of language models, each with a progressively larger number of parameters. The smaller models in the series (e.g., 1 billion and 10 billion parameters) consistently fail to accurately perform multi-step arithmetic calculations. However, the largest model in the series (100 billion parameters) suddenly demonstrates the ability to solve these problems with high accuracy, even though this specific skill was not part of its explicit training objectives. Which of the following statements best evaluates this newly observed arithmetic ability?
Analyzing Model Behavior at Scale
Distinguishing Model Improvements