Host DRAM Bandwidth Bottleneck in CPU Expert Execution
At small batch sizes typical of edge decoding, MoE expert computation is memory-bandwidth bound because each token requires streaming the full weight tensors of its routed experts once. Consumer desktop and laptop platforms connect CPUs through dual-channel DRAM, offering peak memory bandwidths of approximately for dual-channel DDR4 and for DDR5. In contrast, modern discrete GPUs achieve from on-package VRAM. Due to this severe host memory bandwidth bottleneck, executing missed experts purely on the CPU caps decode throughput at a small fraction of the GPU's potential, regardless of the number of available CPU cores.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.1 Edge Serving Bottlenecks and Dynamics - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Decode Cache Misses and Host Bandwidth Bottlenecks - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
Static Expert Placement Inefficiency in Edge MoE Decode
Host DRAM Bandwidth Bottleneck in CPU Expert Execution
Hardware-Specific Trade-Off in Serving MoE Decode Cache Misses
Systems Challenge of Edge MoE Serving
Why do static expert placements capture only a small fraction of routed requests during the decode phase of MoE models?
In hybrid MoE serving systems, static expert allocation to GPU VRAM or host memory takes place dynamically during the decode phase.
Under static expert placement during MoE decode, what state is the PCIe interconnect left in when expert computations miss the GPU cache?
Explain why static expert placement leads to inefficiencies during the MoE decode phase, detailing where expert computations execute and how system hardware resources are impacted.
Host DRAM Bandwidth Bottleneck in CPU Expert Execution
Learn After
Match each concept related to edge MoE execution with its accurate description.
Order the following memory configurations from lowest to highest peak memory bandwidth based on edge system specifications.
Explain why allocating additional CPU cores fails to resolve the decode throughput bottleneck during CPU expert execution.
Hardware-Specific Trade-Off in Serving MoE Decode Cache Misses