Learn Before
Bandwidth-Adaptive Decode and the q* Execution Policy - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
CUDA-Graph-Compatible Device-Side Cache Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Semantic-Aware LRU Expert Caching in MoE Decode
Bandwidth-Adaptive Miss Partitioning in MoE Decode
When decode-time cache misses occur, bandwidth-adaptive execution splits the set of missing experts into two disjoint subsets: a cache-fill set containing experts, and a CPU-execution set containing experts, such that . Experts assigned to are transferred over the PCIe bus into GPU cache slots, evaluated on the GPU, and retained in VRAM for future reuse. Concurrently, experts assigned to execute in place directly from host DRAM by the CPU without modifying GPU cache residency. This concurrent partitioning utilizes the PCIe bus at full bandwidth while simultaneously using residual host memory bandwidth for CPU computation, avoiding GPU idle time and maintaining continuous cache updates.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.3 Adaptive Runtime Policies and Device Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Bandwidth-Adaptive Decode and the q* Execution Policy - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
CUDA-Graph-Compatible Device-Side Cache Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
Semantic-Aware LRU Expert Caching in MoE Decode
Bandwidth-Adaptive Miss Partitioning in MoE Decode
Residual Host Bandwidth Formulation for CPU Expert Execution
Optimal Expert Miss Split Ratio (q* Policy)
Hardware-Specific Trade-Off in Serving MoE Decode Cache Misses
Device-Side Dynamic Cache Control via Data-Represented CUDA Graphs
Single-Pass Top-K LRU Victim Selection
Fused Multi-Bank Expert Transfer via Device-Resident Work Lists
Graph-Resident Heterogeneous CPU-GPU Execution Replay
Semantic-Aware LRU Expert Caching in MoE Decode
Bandwidth-Adaptive Miss Partitioning in MoE Decode
Which type of locality is demonstrated during MoE decoding when consecutive decoding steps within a layer repeatedly activate overlapping or recently utilized experts?
When a newly selected expert is not present in a full GPU VRAM cache during MoE decode, what two cache actions take place to manage expert residency?
How does semantic-aware LRU expert caching structure GPU VRAM allocation compared to traditional static expert placement?
During MoE decoding, what state change occurs within the caching system when an activated expert is already present in GPU VRAM (a cache hit)?
Bandwidth-Adaptive Miss Partitioning in MoE Decode
Learn After
In bandwidth-adaptive execution, how are the missing experts in set partitioned between the cache-fill set and the CPU-execution set ?
True or False: Bandwidth-adaptive execution throttles PCIe bus utilization below peak bandwidth to avoid saturating host memory channels during concurrent decode.
If a decode step encounters missing experts and assigns experts to the cache-fill set, what is the size of the CPU-execution set in terms of and ?
Explain how missing experts are handled in both the cache-fill set and the CPU-execution set during bandwidth-adaptive decode, and explain how this partitioning avoids GPU idle time while supporting future decode steps.
How does the execution of experts assigned to the CPU-execution set affect GPU cache residency?
True or False: Experts assigned to the cache-fill set are transferred to the GPU, evaluated, and then discarded from VRAM rather than retained.
From which specific memory do experts assigned to the CPU-execution set execute in place?
Analyze the bandwidth utilization strategy employed during bandwidth-adaptive miss partitioning. Specifically, discuss how the system balances PCIe bus utilization against residual host memory bandwidth during decode, and explain the architectural trade-offs of this concurrent approach compared to transferring all missing experts over the PCIe bus.
Residual Host Bandwidth Formulation for CPU Expert Execution