Given the scenario below, determine which action would result in a greater reduction in the expected test loss and justify your reasoning by analyzing the components of the provided formula.

Google

Hoffmann et al. (2022) established a precise empirical equation for the Chinchilla scaling law to compute the test loss ($$\mathcal{L}$$) based on the model size ($$N$$) and the dataset size ($$D$$). The formulation is expressed as:

$$ \mathcal{L}(N,D) = \underbrace{\frac{406.4}{N^{0.34}}}_{\text{model scaling}} + \underbrace{\frac{410.7}{D^{0.28}}}_{\text{dataset scaling}} + \underbrace{1.69}_{\text{irreducible error}} $$

This relationship divides the overall loss into three distinct components: a model scaling term, a dataset scaling term, and a baseline irreducible error of $${}1.69$$.

Chinchilla Scaling Law Formula

Optimizing Training Resources

Consider the empirical formula for the Chinchilla scaling law: $$ \mathcal{L}(N,D) = \frac{406.4}{N^{0.34}} + \frac{410.7}{D^{0.28}} + 1.69 $$ If a research team had access to a dataset of infinite size (i.e., as D approaches infinity), what would be the theoretical minimum loss for a model of a fixed size N? Explain how you arrived at your answer by analyzing the components of the formula.

Theoretical Loss Limit with Infinite Data

A research team is using the following empirical formula to guide their training strategy for a large language model, where `L` is the test loss, `N` is the model size, and `D` is the dataset size: $$ \mathcal{L}(N,D) = \frac{406.4}{N^{0.34}} + \frac{410.7}{D^{0.28}} + 1.69 $$ To achieve the most substantial reduction in test loss, which of the following strategies is predicted by this formula to be more effective?

Learn Before

Related