Learn Before
CUDA-Graph-Compatible Device-Side Cache Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Expert Storage Formats and Platform Adaptation - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Device-Side Dynamic Cache Control via Data-Represented CUDA Graphs
Fused Multi-Bank Expert Transfer via Device-Resident Work Lists
To transfer missing expert parameters across PCIe without host synchronization overhead, expert weights are structured so that all parameter banks share identical logical expert-to-slot mappings. Once victim slots are selected on the GPU, a copy work list is generated in device memory containing source and destination indices. This single index list is launched across all weight banks in a single, fused memory transfer kernel of fixed shape. Any unused slots within the fixed-dimension work buffer are masked out using a device-resident valid count, minimizing kernel launch overhead, sustaining maximum PCIe bus saturation, and keeping execution entirely device-resident.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.3 Adaptive Runtime Policies and Device Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
CUDA-Graph-Compatible Device-Side Cache Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.4 Platform Adaptation and Performance Evaluation - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Expert Storage Formats and Platform Adaptation - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
Device-Side Dynamic Cache Control via Data-Represented CUDA Graphs
Single-Pass Top-K LRU Victim Selection
Fused Multi-Bank Expert Transfer via Device-Resident Work Lists
Graph-Resident Heterogeneous CPU-GPU Execution Replay
Semantic-Aware LRU Expert Caching in MoE Decode
Bandwidth-Adaptive Miss Partitioning in MoE Decode
Normalized Expert Bank Representation
FreeToken Weight (FTW) Storage Format
Pure-CPU MoE Fallback Backend
Direct Host Layout Loading with Deferred Memory Pinning
Fused Multi-Bank Expert Transfer via Device-Resident Work Lists
Match each device kernel task to its corresponding role in device-side dynamic cache control.
To avoid host intervention while supporting static CUDA Graphs, variable expert quantities are tracked on the GPU using device-resident ___ counts within fixed-shape buffers.
Why is managing expert caching decisions on the host CPU disadvantageous during Mixture-of-Experts (MoE) serving?
According to the device-side cache control mechanism, what two destinations or indicators can the dedicated device kernel map logical routed identifiers into?
Single-Pass Top-K LRU Victim Selection
Fused Multi-Bank Expert Transfer via Device-Resident Work Lists
Graph-Resident Heterogeneous CPU-GPU Execution Replay
Learn After
Match each component of the device-resident expert transfer mechanism to its primary function.
Order the steps involved in executing a fused multi-bank expert transfer across PCIe without host synchronization.
How should the team redesign the parameter bank layouts and transfer execution to eliminate host synchronization and maximize PCIe bus saturation?
What is the primary operational benefit of launching a single, fused memory transfer kernel across all weight banks rather than dispatching separate transfer kernels per bank?
To enable fused expert transfer across PCIe without host synchronization overhead, each parameter bank must maintain an independent, bank-specific expert-to-slot mapping.
Where is the copy work list containing source and destination indices created and stored during the expert transfer process?
Evaluate the architectural trade-offs of utilizing a fixed-dimension work buffer rather than dynamically resizing the buffer for PCIe expert parameter transfers. In your response, explain how the system handles transfers when fewer experts are needed than the buffer capacity, and describe how this approach preserves device-resident execution.