Analyze the bandwidth utilization strategy employed during bandwidth-adaptive miss partitioning. Specifically, discuss how the system balances PCIe bus utilization against residual host memory bandwidth during decode, and explain the architectural trade-offs of this concurrent approach compared to transferring all missing experts over the PCIe bus.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.3 Adaptive Runtime Policies and Device Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Bandwidth-Adaptive Decode and the q* Execution Policy - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
CUDA-Graph-Compatible Device-Side Cache Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
In bandwidth-adaptive execution, how are the missing experts in set partitioned between the cache-fill set and the CPU-execution set ?
True or False: Bandwidth-adaptive execution throttles PCIe bus utilization below peak bandwidth to avoid saturating host memory channels during concurrent decode.
If a decode step encounters missing experts and assigns experts to the cache-fill set, what is the size of the CPU-execution set in terms of and ?
Explain how missing experts are handled in both the cache-fill set and the CPU-execution set during bandwidth-adaptive decode, and explain how this partitioning avoids GPU idle time while supporting future decode steps.
How does the execution of experts assigned to the CPU-execution set affect GPU cache residency?
True or False: Experts assigned to the cache-fill set are transferred to the GPU, evaluated, and then discarded from VRAM rather than retained.
From which specific memory do experts assigned to the CPU-execution set execute in place?
Analyze the bandwidth utilization strategy employed during bandwidth-adaptive miss partitioning. Specifically, discuss how the system balances PCIe bus utilization against residual host memory bandwidth during decode, and explain the architectural trade-offs of this concurrent approach compared to transferring all missing experts over the PCIe bus.
Residual Host Bandwidth Formulation for CPU Expert Execution