Investigating Training Instability with Mixed-Precision Hardware
Based on the principles of numerical computation in distributed systems, identify and explain two distinct potential causes for the training process failing with a NaN loss value.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Low-Precision Arithmetic Challenges in Distributed Training
Impact of Floating-Point Non-Associativity in Gradient Accumulation
Diagnosing Instability in Large-Scale Model Training
A team training a large model on a distributed system notices a peculiar issue. When they perform a gradient accumulation step by summing gradients from all worker nodes, the final aggregated gradient value on the parameter server slightly differs depending on the order in which the gradients arrive and are summed. The team has verified that all worker nodes are using identical hardware and are calculating their individual gradients correctly. Which of the following best explains this phenomenon?
Investigating Training Instability with Mixed-Precision Hardware