Learn Before
Addressing Training Instability in a Large Language Model
Given the following case study, propose one specific modification to the model's internal structure that is known to improve training stability at a large scale. Explain the mechanism by which your proposed modification helps prevent the observed training failure.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Addressing Training Instability in a Large Language Model
Match each architectural modification with its primary purpose in stabilizing the training of very large language models.
A team training a multi-billion parameter language model observes that the training process frequently fails due to sudden, large spikes in the loss function and exploding gradient values. This instability becomes more pronounced as they increase the model's depth. Which of the following architectural modifications is most specifically designed to counteract this particular problem and improve training stability in very deep models?