In distributed deep learning, transferring data between devices (such as moving gradients from a GPU to the CPU) utilizes the system bus (e.g., PCI-Express), which is a hardware resource distinct from the computational units. Because these resources are separate, frameworks can overlap computation and communication to significantly reduce total execution time. For example, during backpropagation on a minibatch, some parameter gradients become available earlier than others. A system can begin copying the computed gradients over the bus while the GPU concurrently processes the remaining gradients, since copying $$y[i-1]$$ does not conflict with computing $$y[i]$$.

Overlapping Computation and Communication

Moving tensor data across devices severely complicates parallel processing because computational operations must block, or pause, while waiting for the necessary data to be transmitted and received over the system bus. Due to the high baseline overhead of initiating these data transfers, executing numerous small, interspersed copy operations is drastically worse for performance than consolidating data into a single, large transfer operation.

Claude

Transferring tensor data between different hardware devices (such as moving data from the main memory to a GPU) is an exceptionally slow operation, typically much slower than executing the mathematical computations themselves. Deep learning frameworks intentionally require users to explicitly command these transfers rather than performing them automatically under the hood. This design prevents developers from inadvertently writing highly inefficient code where the framework silently copies data back and forth, crashing the program instead to alert the user of the device mismatch.

Learn Before

Related

Learn After