Combined Power Law for LLM Loss with Model and Dataset Size

A research team is deciding between two language model sizes. Model A will have 10 billion parameters, and Model B will have 100 billion parameters. According to the empirical relationship where performance loss (L) is a function of the number of parameters (N), as shown in the formula below, which model should the team choose to achieve a lower final loss, and what is the justification?

$L(N) = \left(\frac{N}{8.8 \times 10^{13}}\right)^{-0.076}$

Interpreting Model Scaling Effects

Interpreting the Model Scaling Formula

Visualizing Empirical Scaling Laws for LLM Loss

Power Law Fit for Test Loss vs. Model and Dataset Size