Combined Power Law for LLM Loss with Model and Dataset Size

A research team observes that as they increase the computational resources (x) used to train a language model, the model's final loss (L) decreases. However, the loss curve begins to flatten out, suggesting it is approaching a minimum value greater than zero and will not improve further, regardless of additional resources. Given the relationship L(x) = ax^b + ε_∞, which component of the formula is responsible for this 'performance floor' phenomenon?

Comparing LLM Training Potential

Evaluating a Model Training Proposal