Deep learning performance depends heavily on the seamless movement of data from durable storage and RAM to the processors (CPUs or GPUs). If data cannot be loaded quickly enough, or if matrices cannot be moved rapidly to the accelerators, the processing elements will starve, creating a major system bottleneck. To achieve optimal performance, systems must efficiently shuffle data and often interleave communication with computation.

Hardware Data Movement Bottlenecks

Within a standard computer architecture for deep learning, most high-performance components—such as the network interface, GPU accelerators, and durable storage—are attached to the CPU across the PCIe bus. In contrast, the main memory (RAM) is directly attached to the CPU, offering massive total bandwidth (e.g., up to $$100$$ GB/s) to ensure data can flow seamlessly into the processor during intensive computations.

Claude

The Peripheral Component Interconnect Express (PCIe) is a dedicated high-speed expansion bus that forms point-to-point connections between the CPU and critical components like GPUs, network interfaces, and storage. It offers massive bandwidth, such as up to $$32$$ GB/s on PCIe 4.0 in a $$16$$-lane slot, with single-digit microsecond latency (e.g., $$5$$ μs). However, PCIe links are a precious resource because processors only support a limited number of lanes. For instance, AMD's EPYC 3 has $$128$$ lanes, Intel's Xeon has up to $$48$$ lanes per chip, while desktop-grade CPUs like the Ryzen 9 and Core i9 have $$20$$ and $$16$$ lanes, respectively. Because GPUs typically require $$16$$ lanes each and must share the bus with other bandwidth-intensive peripherals, the number of GPUs that can connect to the CPU at full speed is inherently limited. To minimize packet overhead during transmission across these limited links, deep learning systems prefer large bulk data transfers.

PCI Express (PCIe) in Deep Learning

Dive into Deep Learning

Hardware Connectivity Architecture in Deep Learning

In multi-GPU systems, central processing units (CPUs) often possess an insufficient number of direct PCIe lanes to establish individual connections with every Graphics Processing Unit (GPU). For example, consumer-grade CPUs may only support $$24$$ lanes, while each GPU typically requires $$16$$ lanes for optimal throughput. To overcome this hardware limitation, systems employ a PCIe multiplexer, or switch. This switch acts as an intermediary hub that facilitates full-bandwidth communication directly between the connected devices. Because each GPU connects to the multiplexer at high speeds (e.g., $$16$$ GB/s on a 16x Gen3 link), the switch enables simultaneous, device-to-device data transfers that are significantly more efficient than routing all communication through the CPU.

Learn Before

Related

Learn After