Learn Before
A team training a multi-billion parameter language model observes that the training process frequently fails due to sudden, large spikes in the loss function and exploding gradient values. This instability becomes more pronounced as they increase the model's depth. Which of the following architectural modifications is most specifically designed to counteract this particular problem and improve training stability in very deep models?
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Addressing Training Instability in a Large Language Model
Match each architectural modification with its primary purpose in stabilizing the training of very large language models.
A team training a multi-billion parameter language model observes that the training process frequently fails due to sudden, large spikes in the loss function and exploding gradient values. This instability becomes more pronounced as they increase the model's depth. Which of the following architectural modifications is most specifically designed to counteract this particular problem and improve training stability in very deep models?