Unified Slot Pool and Decode Seeding in Edge MoE Serving
In edge MoE serving, prefill double buffers and the decode expert cache are allocated from a single shared global slot pool rather than isolated memory partitions. Because both execution phases draw from the same residency pool, the serving engine eliminates the overhead of a dedicated prefill cache and avoids explicit phase-handoff reallocations. Moreover, expert weights already loaded in VRAM that persist across prefill automatically remain in the shared pool, effectively seeding the GPU expert cache for the subsequent latency-sensitive decode phase.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.2 Pipelining and State Caching Mechanisms - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Pipelined Loading via Full-Layer Double Buffering - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
Expert Transfer Bottleneck in Edge MoE Prefill
Pipelined Full-Layer Double Buffering in MoE Prefill
Unified Slot Pool and Decode Seeding in Edge MoE Serving
Why can background transfer of layer begin immediately during the computation of layer when using full-layer double buffering?
If GPU memory is insufficient to allocate two full-layer buffers, the serving engine falls back to on-demand expert loading.
What action do the two full-layer buffers undergo once computation for a layer completes?
Explain how full-layer double buffering overlaps computation with communication during the prefill phase of an MoE model, and describe the rationale behind buffering full layers rather than individual experts.
Match each component of the MoE full-layer double-buffering architecture to its operational role.
Order the operational stages of pipelined full-layer double buffering during MoE prefill from initialization through layer progression.
Explain the timing constraint required for full-layer double buffering to completely hide expert weight communication, and explain why the GPU compute stream is idling at layer boundaries in this scenario.
Unified Slot Pool and Decode Seeding in Edge MoE Serving
Learn After
Match each memory management concept in edge MoE serving to its correct description:
Order the lifecycle stages of expert weights in edge MoE serving as execution transitions across phases in a unified residency structure:
Based on the memory management principles for edge MoE serving, explain how transitioning to a single shared global slot pool remedies these phase-transition latency bottlenecks.