Residual Host Bandwidth Formulation for CPU Expert Execution
In an edge system where direct CPU expert evaluation and direct memory access (DMA) transfers across PCIe draw from the same host DRAM subsystem, a saturated PCIe transfer leaves limited memory bandwidth for CPU processing. If denotes the measured host-side expert-processing bandwidth and denotes the pinned expert-transfer bandwidth across PCIe, the residual host bandwidth available for concurrent in-place CPU expert computation is: When , the CPU can execute missed experts using this residual bandwidth without throttling the saturated PCIe transfer.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.3 Adaptive Runtime Policies and Device Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Bandwidth-Adaptive Decode and the q* Execution Policy - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
Semantic-Aware LRU Expert Caching in MoE Decode
Bandwidth-Adaptive Miss Partitioning in MoE Decode
Residual Host Bandwidth Formulation for CPU Expert Execution
Hardware-Specific Trade-Off in Serving MoE Decode Cache Misses
In bandwidth-adaptive execution, how are the missing experts in set partitioned between the cache-fill set and the CPU-execution set ?
True or False: Bandwidth-adaptive execution throttles PCIe bus utilization below peak bandwidth to avoid saturating host memory channels during concurrent decode.
If a decode step encounters missing experts and assigns experts to the cache-fill set, what is the size of the CPU-execution set in terms of and ?
Explain how missing experts are handled in both the cache-fill set and the CPU-execution set during bandwidth-adaptive decode, and explain how this partitioning avoids GPU idle time while supporting future decode steps.
How does the execution of experts assigned to the CPU-execution set affect GPU cache residency?
True or False: Experts assigned to the cache-fill set are transferred to the GPU, evaluated, and then discarded from VRAM rather than retained.
From which specific memory do experts assigned to the CPU-execution set execute in place?
Analyze the bandwidth utilization strategy employed during bandwidth-adaptive miss partitioning. Specifically, discuss how the system balances PCIe bus utilization against residual host memory bandwidth during decode, and explain the architectural trade-offs of this concurrent approach compared to transferring all missing experts over the PCIe bus.
Residual Host Bandwidth Formulation for CPU Expert Execution
Learn After
Order the steps to determine and utilize residual host bandwidth for CPU expert execution under a saturated PCIe bus.
Calculate the residual host bandwidth (B_R) available for concurrent in-place CPU computation, and explain whether the CPU can execute missed experts concurrently without contending with or throttling the PCIe bus.
Optimal MoE Decode Cache-Fill Count ( Policy)