Learn Before
Considerations for Stabilizing Large-Scale Model Training
To overcome the instability and convergence difficulties encountered when training extremely large pre-trained models, researchers must carefully consider several complex engineering aspects. Key elements to manage include the specific model architecture, the implementation of parallel computation, and the techniques used for parameter initialization.
0
1
Tags
Foundations of Large Language Models
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Learning Rate and Training Time Trade-off in LLMs
Multiple Approaches to Enhance LLM Training Stability
Evaluating a Training Strategy for a Large Model
Architectural Modifications for Trainable LLMs
A research team successfully trains a 1-billion-parameter language model. Encouraged by their results, they scale up the exact same architecture and training setup to a 100-billion-parameter version using a much larger dataset. Midway through the training process, the model's loss value suddenly becomes
NaN(Not a Number), and the training crashes. This happens repeatedly despite restarting from previous checkpoints. Which of the following best explains this phenomenon?A machine learning team is training a very large language model and encounters several issues. Match each observed issue with the most likely underlying factor related to training stability.
Considerations for Stabilizing Large-Scale Model Training
Factors Influencing LLM Training Optimization