Single-Pass Top-K LRU Victim Selection
Standard Least Recently Used (LRU) cache eviction policies typically require scanning the cache metadata once per evicted item, incurring high latency when evicting multiple slots. In edge MoE serving, single-pass victim selection avoids multiple linear scans through a dedicated GPU kernel that identifies the least-recently-used candidate slots across the entire cache in a single pass. When a step encounters cache misses requiring eviction, the miss handler directly consumes the top candidate slots from the pre-discovered list. This ensures that victim discovery overhead remains strictly constant at one pass per layer regardless of the number of misses realized at runtime.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.3 Adaptive Runtime Policies and Device Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
CUDA-Graph-Compatible Device-Side Cache Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
Device-Side Dynamic Cache Control via Data-Represented CUDA Graphs
Single-Pass Top-K LRU Victim Selection
Fused Multi-Bank Expert Transfer via Device-Resident Work Lists
Graph-Resident Heterogeneous CPU-GPU Execution Replay
Semantic-Aware LRU Expert Caching in MoE Decode
Bandwidth-Adaptive Miss Partitioning in MoE Decode
Match each device kernel task to its corresponding role in device-side dynamic cache control.
To avoid host intervention while supporting static CUDA Graphs, variable expert quantities are tracked on the GPU using device-resident ___ counts within fixed-shape buffers.
Why is managing expert caching decisions on the host CPU disadvantageous during Mixture-of-Experts (MoE) serving?
According to the device-side cache control mechanism, what two destinations or indicators can the dedicated device kernel map logical routed identifiers into?
Single-Pass Top-K LRU Victim Selection
Fused Multi-Bank Expert Transfer via Device-Resident Work Lists
Graph-Resident Heterogeneous CPU-GPU Execution Replay
Learn After
In edge MoE serving, at what execution granularity does single-pass victim selection maintain a strictly constant victim discovery overhead?
When a step encounters q cache misses (where q <= K), the miss handler must perform q additional linear scans to select the final eviction slots from the pre-discovered candidates.
Why do standard Least Recently Used (LRU) cache eviction policies incur high latency when evicting multiple slots?
Analyze the role of the pre-discovered candidate list in single-pass LRU victim selection during edge MoE serving. Explain how the miss handler processes runtime cache misses using this list, and explain why the condition is essential to the policy's operational guarantees.