Learn Before
Chinchilla
Chinchilla is a 70-billion-parameter language model that inherits the architecture and compute budget of Gopher but trains on a significantly larger dataset of 1.4 trillion tokens. It outperforms Gopher by placing greater emphasis on the number of training tokens rather than the sheer number of model parameters.
0
1
Tags
D2L
Dive into Deep Learning @ D2L
Related
Chinchilla
Chinchilla Scaling Law Formula
A research team is training a large language model and observes that the model's performance, measured by test loss, seems to be primarily limited by the number of model parameters rather than the amount of training data. According to the principle that models test loss as a sum of two separate terms—one that decreases as model size grows and another that decreases as dataset size grows—which of the following actions would most effectively reduce the test loss in this specific situation?
Critique of a Model Training Strategy
Analyzing Performance Limits in Language Models
Chinchilla