Formula

Optimal Expert Miss Split Ratio (q* Policy)

To minimize exposed decode latency when serving mm missing experts of size SS bytes each, the execution times of the concurrent PCIe cache-fill branch and CPU execution branch are balanced. The cache-fill duration is Tfill(q)qSBPT_{\text{fill}}(q) \approx \frac{q S}{B_P}, and the CPU execution duration on residual bandwidth is Tcpu(mq)(mq)SBHBPT_{\text{cpu}}(m - q) \approx \frac{(m - q) S}{B_H - B_P}. Equating the execution times of the two concurrent branches yields:

qmqBPBHBP\frac{q}{m - q} \approx \frac{B_P}{B_H - B_P}

Solving for the optimal cache-fill count qq^* produces the closed-form policy:

qmBPBHq^* \approx m \cdot \frac{B_P}{B_H}

In runtime implementation, qq^* is rounded to an integer with a minimum of at least one fill retained to ensure continuous cache warming, dynamically degenerating to pure on-demand GPU caching as BHBPB_H \to B_P.

0

1

Updated 2026-09-07

Tags

Prep Sessions

Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor

Ch.3 Adaptive Runtime Policies and Device Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor

Bandwidth-Adaptive Decode and the q* Execution Policy - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor