Cross-Hardware MoE Serving across Memory and Interconnect Tiers
The practical performance of hybrid MoE execution across consumer systems depends directly on the relative balance between host memory bandwidth () and PCIe transfer bandwidth (). On dual-channel consumer desktop platforms, CPU-side expert execution encounters a memory bandwidth ceiling (approximately ), causing CPU-bound baselines to lose 20% or more of their generation throughput compared to multi-channel server configurations. By contrast, bandwidth-adaptive serving dynamically routes miss handling toward PCIe-driven GPU cache fills when interconnect bandwidth is high relative to host RAM, allowing consumer desktops to preserve up to 96% of multi-channel server throughput. Across heterogeneous form factors—from an 8 GB laptop GPU over PCIe 4.0 x8 to a workstation GPU running 700B+ models—adapting miss distribution to measured bandwidth ratios delivers consistent acceleration over static offloading schemes.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.4 Platform Adaptation and Performance Evaluation - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Agentic Workload Serving and Cross-Hardware Performance - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
Decode Throughput Stability in Agentic Serving
Tail Time-to-First-Token as an Agentic Availability Boundary
Cross-Hardware MoE Serving across Memory and Interconnect Tiers
Pipelined Full-Layer Double Buffering in MoE Prefill
Time to First Token (TTFT)
Tokens Per Second (TPS)
Throughput
During MoE decoding, under what specific circumstance does the system need to decide between transferring expert weights across PCIe and computing directly on the CPU?
When an active expert is computed entirely on the CPU during a decode cache miss, the system still receives the benefit of future cache hits from a GPU VRAM cache fill.
Under what condition does exclusively transferring expert weights across PCIe to the GPU leave host DRAM bandwidth and CPU cores idle?
Analyze how the hardware differences between an LPDDR5 laptop and a high-speed PCIe desktop affect the strategy for serving MoE decode cache misses. Ground your analysis in how host memory bandwidth and PCIe transfer bandwidth interact across these platforms.
Why can the optimal strategy for dividing MoE decode cache miss handling between CPU compute and PCIe transfers not be statically predetermined across all systems?
When an active expert is computed entirely on the CPU during an MoE decode cache miss, which hardware interconnect is left idle?
Cross-Hardware MoE Serving across Memory and Interconnect Tiers
Learn After
What memory bandwidth ceiling do dual-channel consumer desktop platforms encounter during CPU-side expert execution, leading to a loss of 20% or more in generation throughput compared to multi-channel servers?
Static offloading schemes dynamically adjust expert miss distribution according to the measured ratio between host memory bandwidth () and PCIe transfer bandwidth ().
What percentage of multi-channel server throughput can consumer desktops preserve when utilizing bandwidth-adaptive serving rather than CPU-bound baselines?
Explain how bandwidth-adaptive serving overcomes the CPU memory bandwidth bottleneck on consumer desktop architectures during hybrid MoE execution.