Learn Before
In edge MoE serving, at what execution granularity does single-pass victim selection maintain a strictly constant victim discovery overhead?
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.3 Adaptive Runtime Policies and Device Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
CUDA-Graph-Compatible Device-Side Cache Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
In edge MoE serving, at what execution granularity does single-pass victim selection maintain a strictly constant victim discovery overhead?
When a step encounters q cache misses (where q <= K), the miss handler must perform q additional linear scans to select the final eviction slots from the pre-discovered candidates.
Why do standard Least Recently Used (LRU) cache eviction policies incur high latency when evicting multiple slots?
Analyze the role of the pre-discovered candidate list in single-pass LRU victim selection during edge MoE serving. Explain how the miss handler processes runtime cache misses using this list, and explain why the condition is essential to the policy's operational guarantees.