During the prefill phase of inference, processing a prompt involves hundreds to thousands of tokens per layer. Although each individual token routes to only a sparse subset of $k$ experts, the union of routed tokens across the entire prompt typically activates nearly the entire expert set in every layer. In an edge deployment where the full model exceeds available GPU memory (VRAM), almost the complete expert pool must be streamed over the CPU–GPU interconnect (e.g., PCIe) during each prefill pass. Consequently, prefill transfer time scales with the total parameter footprint of the model rather than the sparse active path, introducing significant multi-second I/O latency and leaving the GPU idle while weights are fetched on demand.

Expert Transfer Bottleneck in Edge MoE Prefill

To hide massive expert weight movement behind computation during the prefill phase, serving systems can employ full-layer double buffering. Because prompt processing activates nearly the entire expert set across each layer, the system allocates two full-layer buffers in GPU memory from the global slot pool rather than fetching individual experts on demand. While the GPU computes the routed experts for layer $l$ using the active buffer, a dedicated transfer stream concurrently pre-fetches the entire expert set of layer $l + 1$ over PCIe into the alternate buffer. Loading the complete layer allows transfer to begin immediately in the background before the router determines token routing for that layer. Once layer computation completes, the buffers swap roles. If GPU memory is insufficient to allocate two full layers, the engine falls back to on-demand expert loading to prevent memory oversubscription.

Pipelined Full-Layer Double Buffering in MoE Prefill

In edge MoE serving, prefill double buffers and the decode expert cache are allocated from a single shared global slot pool rather than isolated memory partitions. Because both execution phases draw from the same residency pool, the serving engine eliminates the overhead of a dedicated prefill cache and avoids explicit phase-handoff reallocations. Moreover, expert weights already loaded in VRAM that persist across prefill automatically remain in the shared pool, effectively seeding the GPU expert cache for the subsequent latency-sensitive decode phase.

Learn Before

Related

Learn After