Google

Emergent abilities are novel capabilities that Large Language Models exhibit as the scale of their training increases. These abilities are not explicitly designed or programmed but rather appear as a byproduct of scaling, often unexpectedly.

Emergent Abilities in LLMs

The manifestation of emergent abilities in Large Language Models provides compelling evidence for the effectiveness of scaled training. The fact that new, unprogrammed capabilities appear simply by increasing the scale of the model and data reinforces the value of this approach for enhancing LLM performance.

Emergent Abilities as Evidence for the Efficacy of Scaled Training

Analyze the following scenario and determine whether the new capability demonstrated by the 'Innovate-100B' model is an example of an emergent ability. Justify your answer based on the defining characteristics of such abilities.

Evaluating a New Model Capability

A large language model is trained exclusively on a vast corpus of text from encyclopedias and historical documents. After training, it demonstrates two new skills: (1) It can accurately answer factual questions about historical events mentioned in its training data. (2) It can write short, rhyming poems about those same historical events, despite having no poetry in its training set. Which of these two skills is a better example of an emergent ability, and why? Justify your reasoning.

Distinguishing Learned vs. Emergent Skills

A research team trains a series of language models, each with an increasing number of parameters, on the same large dataset. They evaluate each model on several different tasks. Which of the following outcomes would be the clearest example of an emergent ability?

Learn Before

Related