Learn Before
Decode Cache Misses and Host Bandwidth Bottlenecks - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Bandwidth-Adaptive Decode and the q* Execution Policy - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Host DRAM Bandwidth Bottleneck in CPU Expert Execution
Hardware-Specific Trade-Off in Serving MoE Decode Cache Misses
When an active expert is not present in GPU VRAM during decoding, the system can either transfer its weights across PCIe to execute on the GPU, or compute it directly on the CPU where the weights reside in host RAM. Neither extreme is universally optimal: exclusively moving weights over PCIe leaves available host DRAM bandwidth and CPU cores idle if host memory can supply more data than the bus can move; conversely, computing entirely on the CPU leaves the PCIe interconnect idle and forfeits future cache hits that a VRAM cache fill provides. Because the balance between host memory bandwidth and PCIe transfer bandwidth varies widely across machines (such as LPDDR5 laptops versus high-speed PCIe desktops), the optimal division of miss handling cannot be statically predetermined and must be tuned to specific hardware resources.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.1 Edge Serving Bottlenecks and Dynamics - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Decode Cache Misses and Host Bandwidth Bottlenecks - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.3 Adaptive Runtime Policies and Device Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Bandwidth-Adaptive Decode and the q* Execution Policy - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
Static Expert Placement Inefficiency in Edge MoE Decode
Host DRAM Bandwidth Bottleneck in CPU Expert Execution
Hardware-Specific Trade-Off in Serving MoE Decode Cache Misses
Systems Challenge of Edge MoE Serving
Semantic-Aware LRU Expert Caching in MoE Decode
Bandwidth-Adaptive Miss Partitioning in MoE Decode
Residual Host Bandwidth Formulation for CPU Expert Execution
Optimal Expert Miss Split Ratio (q* Policy)
Hardware-Specific Trade-Off in Serving MoE Decode Cache Misses
Match each concept related to edge MoE execution with its accurate description.
Order the following memory configurations from lowest to highest peak memory bandwidth based on edge system specifications.
Explain why allocating additional CPU cores fails to resolve the decode throughput bottleneck during CPU expert execution.
Hardware-Specific Trade-Off in Serving MoE Decode Cache Misses
Learn After
During MoE decoding, under what specific circumstance does the system need to decide between transferring expert weights across PCIe and computing directly on the CPU?
When an active expert is computed entirely on the CPU during a decode cache miss, the system still receives the benefit of future cache hits from a GPU VRAM cache fill.
Under what condition does exclusively transferring expert weights across PCIe to the GPU leave host DRAM bandwidth and CPU cores idle?
Analyze how the hardware differences between an LPDDR5 laptop and a high-speed PCIe desktop affect the strategy for serving MoE decode cache misses. Ground your analysis in how host memory bandwidth and PCIe transfer bandwidth interact across these platforms.
Why can the optimal strategy for dividing MoE decode cache miss handling between CPU compute and PCIe transfers not be statically predetermined across all systems?
When an active expert is computed entirely on the CPU during an MoE decode cache miss, which hardware interconnect is left idle?
Cross-Hardware MoE Serving across Memory and Interconnect Tiers