Motivation for Continued Scaling of LLMs

Example of Emergent Abilities Study (Wei et al., 2022b)

A research lab trains a series of language models, each with a progressively larger number of parameters. The smaller models in the series (e.g., 1 billion and 10 billion parameters) consistently fail to accurately perform multi-step arithmetic calculations. However, the largest model in the series (100 billion parameters) suddenly demonstrates the ability to solve these problems with high accuracy, even though this specific skill was not part of its explicit training objectives. Which of the following statements best evaluates this newly observed arithmetic ability?

Analyzing Model Behavior at Scale

Distinguishing Model Improvements