Emergent Capabilities of LLMs
As Large Language Models are trained at progressively larger scales, they often develop novel abilities that were not present in smaller versions of the models. These unforeseen skills, which appear to be a direct consequence of the increased scale of training, are known as emergent capabilities.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Capabilities of Scaled LLMs
Emergent Abilities in LLMs
Emergent Capabilities of LLMs
A research lab is attempting to develop a language model that exhibits a complex, unforeseen skill, such as advanced causal reasoning, which is absent in their current, smaller models. They understand that such 'emergent abilities' are not explicitly programmed but appear as a result of scale. Given limited resources, which of the following approaches represents the most effective strategy for achieving this goal?
Analysis of Competing LLM Scaling Strategies
Match each key dimension of scaling a language model to the description that best explains how it contributes to the potential for developing new, advanced capabilities.
Learn After
Motivation for Continued Scaling of LLMs
Example of Emergent Abilities Study (Wei et al., 2022b)
A research lab trains a series of language models, each with a progressively larger number of parameters. The smaller models in the series (e.g., 1 billion and 10 billion parameters) consistently fail to accurately perform multi-step arithmetic calculations. However, the largest model in the series (100 billion parameters) suddenly demonstrates the ability to solve these problems with high accuracy, even though this specific skill was not part of its explicit training objectives. Which of the following statements best evaluates this newly observed arithmetic ability?
Analyzing Model Behavior at Scale
Distinguishing Model Improvements