Diagnosing Low-Precision Training Failures

A team is performing distributed training of a large model using an 8-bit floating-point format for speed. They observe that while the training process is stable on most of their compute nodes, a specific group of nodes consistently fails, with the model's weights rapidly becoming infinite values. Which computational challenge is the most direct and likely cause of this specific failure mode?

A research team is training a large model across a heterogeneous cluster of computing devices from different manufacturers. They are using a low-precision 8-bit numerical format to accelerate the process. They observe that when they run the exact same training job with the same initial random seed, the final model parameters diverge slightly depending on which specific set of devices was allocated for the run. The training does not crash, and no error messages are generated. What is the most probable cause for this observed divergence?