Power Law Formula for LLM Loss

Improved Power Law for LLM Loss with Irreducible Error

A research team is developing a series of language models. They systematically increase the number of model parameters and measure the final test loss for each model. They observe a consistent trend: as the number of parameters grows, the test loss steadily decreases. However, the amount of improvement (loss reduction) becomes progressively smaller for each subsequent increase in parameters. Which of the following mathematical forms would be the most straightforward initial choice to model this observed relationship between model size and loss?

Modeling LLM Performance with Power Laws

Modeling LLM Performance Trends