University of Michigan - Ann Arbor

Predictable scaling relies on developing deep learning infrastructure and optimization methods that ensure model behavior follows consistent mathematical trajectories across multiple orders of magnitude. In the training of GPT-4, aspects of final performance—such as codebase next-word prediction loss and coding problem pass rates—were accurately predicted prior to full training by fitting power-law scaling functions to smaller models trained with as little as 1/1,000th the compute budget.

Predictable Scaling in GPT-4

What foundational technical elements are required to ensure that model behavior follows consistent mathematical trajectories across multiple orders of magnitude?

In the training of GPT-4, what was the smallest compute budget—expressed as a fraction of the full budget—used on smaller models to accurately predict aspects of final performance?

Learn Before

Related