Static Expert Placement Inefficiency in Edge MoE Decode
During the decode phase of Mixture-of-Experts (MoE) models, token routing dynamically shifts at every generation step and varies significantly across different tasks. Existing hybrid serving systems statically allocate expert weights to either GPU VRAM or host memory at initialization or during prefill—for instance, by pinning an estimated "hot" set of experts on the GPU. Because runtime routing changes unpredictably, static placements capture only a small fraction of real routed requests. As a result, the vast majority of expert computations miss the GPU cache and are forced onto the CPU, leaving both the GPU compute cores and the PCIe interconnect idle.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.1 Edge Serving Bottlenecks and Dynamics - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Decode Cache Misses and Host Bandwidth Bottlenecks - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Learn After
Why do static expert placements capture only a small fraction of routed requests during the decode phase of MoE models?
In hybrid MoE serving systems, static expert allocation to GPU VRAM or host memory takes place dynamically during the decode phase.
Under static expert placement during MoE decode, what state is the PCIe interconnect left in when expert computations miss the GPU cache?
Explain why static expert placement leads to inefficiencies during the MoE decode phase, detailing where expert computations execute and how system hardware resources are impacted.
Host DRAM Bandwidth Bottleneck in CPU Expert Execution