When a single computational device is insufficient for deep learning optimization, systems rely on networks and buses to synchronize processing by transferring data. The selection of these interconnects depends on design parameters such as bandwidth, cost, distance, and flexibility. For example, while WiFi is inexpensive and flexible, its mediocre bandwidth and latency make it wholly unsuitable for machine learning clusters. Instead, distributed deep learning environments utilize high-speed connections—ranging from standard $1$ GBit/s Ethernet to $100$ GBit/s interconnects in cloud instances—to efficiently synchronize model parameters across multiple systems.