Learn Before
Connecting Model Scale and Architectural Design
A machine learning team successfully trained a 1-billion-parameter language model using a standard network architecture. When they scale the exact same architecture up to 100 billion parameters and begin training with a proportionally larger dataset, they find the training process repeatedly fails. Based on the principles of large-scale model training, explain the most likely reason for this discrepancy in training stability between the two model sizes.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Evaluating a Training Strategy for a New Large Model
Layer Normalization in Transformers
A research team is training a very deep language model based on a standard network design. They observe that as they increase the model's depth, the training process frequently fails with loss values suddenly becoming invalid (NaN). This forces them to restart training repeatedly. Which of the following architectural changes is most specifically designed to mitigate this kind of deep-network training instability?
Rationale for Architectural Changes in Large-Scale Models
Connecting Model Scale and Architectural Design
Omission of Bias Terms in LLM Affine Transformations