Learn Before
Distinguishing Model Improvements
A team of AI researchers is developing two language models.
- Model A: As they increase the model's size from 1 million to 1 billion parameters, its ability to generate grammatically correct sentences improves steadily and predictably.
- Model B: After increasing its size from 10 billion to 100 billion parameters, the model, which previously could only translate between English and Spanish, suddenly gains the ability to write simple, functional computer programs in Python, a skill it was not explicitly trained for.
Based on these descriptions, which model's new skill is an example of an emergent capability? Justify your answer by explaining the key characteristics that define such a capability.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Motivation for Continued Scaling of LLMs
Example of Emergent Abilities Study (Wei et al., 2022b)
A research lab trains a series of language models, each with a progressively larger number of parameters. The smaller models in the series (e.g., 1 billion and 10 billion parameters) consistently fail to accurately perform multi-step arithmetic calculations. However, the largest model in the series (100 billion parameters) suddenly demonstrates the ability to solve these problems with high accuracy, even though this specific skill was not part of its explicit training objectives. Which of the following statements best evaluates this newly observed arithmetic ability?
Analyzing Model Behavior at Scale
Distinguishing Model Improvements