Power Law Formulation for LLM Loss

A research team observes the performance of a new model as they increase the amount of training data. They plot the test error and notice a distinct pattern: the error first decreases, then temporarily increases, before starting to decrease again. When attempting to create a mathematical formula to predict this error curve, which approach would be most appropriate?

Modeling an Unconventional Learning Curve

Evaluating Modeling Approaches for LLM Learning Curves