Evaluating a Language Model Pre-training Strategy

A research team is planning to pre-train a new language model with a fixed computational budget. One senior researcher argues, 'Instead of just using more data with our current simple masked language modeling objective, we should dedicate a significant portion of our budget to developing and implementing a novel, more complex pre-training task. This complexity is the key to unlocking better performance.' Based on the major findings from large-scale model training, which of the following statements provides the most accurate evaluation of this researcher's argument?

Prioritizing Pre-training Efforts