Host DRAM Bandwidth Bottleneck in CPU Expert Execution
At small batch sizes typical of edge decoding, Mixture-of-Experts (MoE) expert computation is memory-bandwidth-bound because each token requires streaming the full weight tensors of its routed experts once. Consumer CPUs connected through dual-channel DRAM have peak memory bandwidths of approximately with DDR4 and with DDR5, whereas modern discrete GPUs achieve from on-package VRAM. This host-memory bandwidth constraint limits the throughput of missed experts executed entirely on the CPU to a small fraction of the GPU's potential, regardless of the number of available CPU cores.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.1 Edge Serving Bottlenecks and Dynamics - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Decode Cache Misses and Host Bandwidth Bottlenecks - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
Static Expert Placement Inefficiency in Edge MoE Decode
Hardware-Specific Trade-Off in Serving MoE Decode Cache Misses
Host DRAM Bandwidth Bottleneck in CPU Expert Execution
Why do static expert placements capture only a small fraction of routed requests during the decode phase of MoE models?
In hybrid MoE serving systems, static expert allocation to GPU VRAM or host memory takes place dynamically during the decode phase.
Under static expert placement during MoE decode, what state is the PCIe interconnect left in when expert computations miss the GPU cache?
Explain why static expert placement leads to inefficiencies during the MoE decode phase, detailing where expert computations execute and how system hardware resources are impacted.
Host DRAM Bandwidth Bottleneck in CPU Expert Execution
Learn After
Match each concept related to edge MoE execution with its accurate description.
Order the following memory configurations from lowest to highest peak memory bandwidth based on edge system specifications.
Explain why allocating additional CPU cores fails to resolve the decode throughput bottleneck during CPU expert execution.
Hardware-Specific Trade-Off in Serving MoE Decode Cache Misses