Learn Before
A team is training an exceptionally deep transformer-based language model and observes that the training process is highly unstable, with loss values fluctuating wildly and sometimes resulting in non-numeric values (NaNs). This suggests that the gradients are either exploding or vanishing as they propagate through the numerous layers. Which of the following architectural modifications is most specifically designed to address this type of instability in very deep networks?
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Diagnosing Instability in Large-Scale Model Training
A team is training an exceptionally deep transformer-based language model and observes that the training process is highly unstable, with loss values fluctuating wildly and sometimes resulting in non-numeric values (NaNs). This suggests that the gradients are either exploding or vanishing as they propagate through the numerous layers. Which of the following architectural modifications is most specifically designed to address this type of instability in very deep networks?
Prioritizing Architectural Modifications for Training Stability