Diagnosing Instability in Large-Scale Model Training

A team is training an exceptionally deep transformer-based language model and observes that the training process is highly unstable, with loss values fluctuating wildly and sometimes resulting in non-numeric values (NaNs). This suggests that the gradients are either exploding or vanishing as they propagate through the numerous layers. Which of the following architectural modifications is most specifically designed to address this type of instability in very deep networks?

Prioritizing Architectural Modifications for Training Stability