Graph-Resident Heterogeneous CPU-GPU Execution Replay
Heterogeneous serving architectures that split computation across CPU cores and GPU streaming multiprocessors typically suffer from per-token runtime scheduling and synchronization latency. Graph-resident heterogeneous execution integrates both execution branches into a unified, statically captured CUDA Graph. For each decode batch size, the system pre-allocates stable pinned input/output buffers and persistent task descriptors. The graph statically bundles device-to-host tensor transfers, a host-function submit node triggering persistent C++ worker threads, concurrent GPU kernel evaluations, a synchronization barrier, and the final host-to-device result copy. During generation, graph replay repeatedly invokes the complete heterogeneous pipeline without triggering per-token Python dispatch.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.3 Adaptive Runtime Policies and Device Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
CUDA-Graph-Compatible Device-Side Cache Execution - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
Device-Side Dynamic Cache Control via Data-Represented CUDA Graphs
Single-Pass Top-K LRU Victim Selection
Fused Multi-Bank Expert Transfer via Device-Resident Work Lists
Graph-Resident Heterogeneous CPU-GPU Execution Replay
Semantic-Aware LRU Expert Caching in MoE Decode
Bandwidth-Adaptive Miss Partitioning in MoE Decode
Match each device kernel task to its corresponding role in device-side dynamic cache control.
To avoid host intervention while supporting static CUDA Graphs, variable expert quantities are tracked on the GPU using device-resident ___ counts within fixed-shape buffers.
Why is managing expert caching decisions on the host CPU disadvantageous during Mixture-of-Experts (MoE) serving?
According to the device-side cache control mechanism, what two destinations or indicators can the dedicated device kernel map logical routed identifiers into?
Single-Pass Top-K LRU Victim Selection
Fused Multi-Bank Expert Transfer via Device-Resident Work Lists
Graph-Resident Heterogeneous CPU-GPU Execution Replay
Learn After
What mechanism in a graph-resident heterogeneous pipeline allows CPU execution to be triggered without per-token runtime scheduling?
Graph replay requires per-token Python dispatch to coordinate synchronization barriers between CPU and GPU tasks.
What resources must the system pre-allocate for each decode batch size to support graph-resident heterogeneous execution?
Describe the sequence of operations statically bundled within a unified CUDA Graph for graph-resident heterogeneous CPU-GPU execution replay.