Optimal Expert Miss Split Ratio (q* Policy)
To minimize exposed decode latency when serving missing experts of size bytes each, the execution times of the concurrent PCIe cache-fill branch and CPU execution branch are balanced. The cache-fill duration is , and the CPU execution duration on residual bandwidth is . Equating the execution times of the two concurrent branches yields:
Solving for the optimal cache-fill count produces the closed-form policy:
In runtime implementation, is rounded to an integer with a minimum of at least one fill retained to ensure continuous cache warming, dynamically degenerating to pure on-demand GPU caching as .
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.3 Adaptive Runtime Policies and Device Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Bandwidth-Adaptive Decode and the q* Execution Policy - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
Semantic-Aware LRU Expert Caching in MoE Decode
Bandwidth-Adaptive Miss Partitioning in MoE Decode
Residual Host Bandwidth Formulation for CPU Expert Execution
Optimal Expert Miss Split Ratio (q* Policy)
Hardware-Specific Trade-Off in Serving MoE Decode Cache Misses
Order the steps to determine and utilize residual host bandwidth for CPU expert execution under a saturated PCIe bus.
Calculate the residual host bandwidth (B_R) available for concurrent in-place CPU computation, and explain whether the CPU can execute missed experts concurrently without contending with or throttling the PCIe bus.
Optimal Expert Miss Split Ratio (q* Policy)
Learn After
A system encounters missing experts during decode, with PCIe bandwidth and host bandwidth . Based on the optimal execution policy, how many missing experts should be routed to the concurrent PCIe cache-fill branch?
In the runtime implementation of the execution policy, a minimum of at least one cache fill is retained to ensure continuous cache warming.
Under the execution policy, what behavior does the system dynamically degenerate to as the host bandwidth approaches PCIe bandwidth ()?
Explain the mathematical derivation used to determine the optimal expert cache-fill count for serving missing experts of size . Include the expressions for both concurrent branches, how they are equated, and the resulting closed-form policy.