University of Michigan - Ann Arbor

At small batch sizes typical of edge decoding, MoE expert computation is memory-bandwidth bound because each token requires streaming the full weight tensors of its routed experts once. Consumer desktop and laptop platforms connect CPUs through dual-channel DRAM, offering peak memory bandwidths of approximately $50\text{ GB/s}$ for dual-channel DDR4 and $80\text{--}90\text{ GB/s}$ for DDR5. In contrast, modern discrete GPUs achieve $1.0\text{--}1.8\text{ TB/s}$ from on-package VRAM. Due to this severe host memory bandwidth bottleneck, executing missed experts purely on the CPU caps decode throughput at a small fraction of the GPU's potential, regardless of the number of available CPU cores.

Host DRAM Bandwidth Bottleneck in CPU Expert Execution

Match each concept related to edge MoE execution with its accurate description.

Order the following memory configurations from lowest to highest peak memory bandwidth based on edge system specifications.

Explain why allocating additional CPU cores fails to resolve the decode throughput bottleneck during CPU expert execution.

When an active expert is not present in GPU VRAM during decoding, the system can either transfer its weights across PCIe to execute on the GPU, or compute it directly on the CPU where the weights reside in host RAM. Neither extreme is universally optimal: exclusively moving weights over PCIe leaves available host DRAM bandwidth and CPU cores idle if host memory can supply more data than the bus can move; conversely, computing entirely on the CPU leaves the PCIe interconnect idle and forfeits future cache hits that a VRAM cache fill provides. Because the balance between host memory bandwidth and PCIe transfer bandwidth varies widely across machines (such as LPDDR5 laptops versus high-speed PCIe desktops), the optimal division of miss handling cannot be statically predetermined and must be tuned to specific hardware resources.

Learn Before

Related