Device-Side Dynamic Cache Control via Data-Represented CUDA Graphs
In Mixture-of-Experts serving, expert caching is inherently dynamic because missing expert identities, transfer batch counts, and eviction slots vary at every generation step. Managing these decisions on the host CPU would force synchronous CPU–GPU device synchronizations at each MoE layer, destroying execution pipelining. To eliminate host intervention while retaining compatibility with static CUDA Graph capture, all routing-dependent decisions are executed entirely on the GPU and encoded as device data within fixed-shape buffers and device-resident valid counts. A dedicated device kernel deduplicates routed expert IDs, checks residency status, computes the bandwidth-proportional fetch quota , marks eviction victims, and maps logical routed identifiers into physical VRAM slots or CPU-execution flags without synchronizing back to the host.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.3 Adaptive Runtime Policies and Device Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
CUDA-Graph-Compatible Device-Side Cache Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
Device-Side Dynamic Cache Control via Data-Represented CUDA Graphs
Single-Pass Top-K LRU Victim Selection
Fused Multi-Bank Expert Transfer via Device-Resident Work Lists
Graph-Resident Heterogeneous CPU-GPU Execution Replay
Semantic-Aware LRU Expert Caching in MoE Decode
Bandwidth-Adaptive Miss Partitioning in MoE Decode
Learn After
Match each device kernel task to its corresponding role in device-side dynamic cache control.
To avoid host intervention while supporting static CUDA Graphs, variable expert quantities are tracked on the GPU using device-resident ___ counts within fixed-shape buffers.
Why is managing expert caching decisions on the host CPU disadvantageous during Mixture-of-Experts (MoE) serving?
According to the device-side cache control mechanism, what two destinations or indicators can the dedicated device kernel map logical routed identifiers into?
Single-Pass Top-K LRU Victim Selection
Fused Multi-Bank Expert Transfer via Device-Resident Work Lists
Graph-Resident Heterogeneous CPU-GPU Execution Replay