Combined Power Law for LLM Loss with Model and Dataset Size

Predicting LLM Performance Based on Dataset Size

A research team observes that their language model's loss (L) decreases as the training dataset size (D) increases, following the specific power law: $L(D) = \left(\frac{D}{C}\right)^{-\alpha}$ where C is a large constant and the exponent α is a small positive number (e.g., 0.095). Based on this mathematical relationship, what is the most significant implication for the team as they consider scaling up their training data from an already very large starting point?

Calculating Loss Reduction from Increased Dataset Size

Power Law Fit for Test Loss vs. Model and Dataset Size