Impact of Floating-Point Non-Associativity in Gradient Accumulation

A team is training a large model using a distributed setup where each node computes gradients in a 16-bit floating-point format to save memory and improve speed. The main copy of the model's parameters is maintained in a more stable 32-bit floating-point format. Before updating these main parameters, the 16-bit gradients from all nodes are collected and summed together. Why is it standard practice to perform this summation into a 32-bit buffer before applying the final update?

Diagnosing Model Divergence in Distributed Training

A research team is training a large model on a distributed system using a low-precision floating-point format for gradient calculations. They run two identical experiments, with the only difference being how the gradients from different compute nodes are summed before the model update. In Experiment A, the gradients are always summed in a fixed, deterministic order. In Experiment B, the summation order varies unpredictably in each training step due to network latency. What is the most likely outcome when comparing the final accumulated gradient values at each step between the two experiments?