Learning Rate and Training Time Trade-off in LLMs

Multiple Approaches to Enhance LLM Training Stability

Evaluating a Training Strategy for a Large Model

Architectural Modifications for Trainable LLMs

A research team successfully trains a 1-billion-parameter language model. Encouraged by their results, they scale up the exact same architecture and training setup to a 100-billion-parameter version using a much larger dataset. Midway through the training process, the model's loss value suddenly becomes NaN (Not a Number), and the training crashes. This happens repeatedly despite restarting from previous checkpoints. Which of the following best explains this phenomenon?

A machine learning team is training a very large language model and encounters several issues. Match each observed issue with the most likely underlying factor related to training stability.

Considerations for Stabilizing Large-Scale Model Training

Factors Influencing LLM Training Optimization