University of Michigan - Ann Arbor

On consumer edge platforms where the full expert pool cannot be locked in pinned memory or registered for Direct Memory Access (DMA)—such as under restrictive operating systems or driver environments—the serving system provides a pure-CPU MoE fallback backend. Under this fallback mode, expert parameters reside in standard pageable host RAM, and all routed expert computations are performed entirely on the CPU. Non-expert model layers remain resident on the GPU, so that only activation-sized tensor inputs, routing metadata, and aggregated expert outputs are transferred across the CPU–GPU bus. While trading off peak PCIe transfer bandwidth, this execution strategy ensures universal deployability on systems incapable of supporting pinned DMA transfers.

Pure-CPU MoE Fallback Backend

In the pure-CPU MoE fallback backend, what memory type hosts the expert parameters?

The pure-CPU MoE fallback backend achieves higher peak PCIe transfer bandwidth than backends utilizing pinned DMA transfers.

What platform or operating system limitations trigger the serving engine to activate the pure-CPU MoE fallback backend?

Describe the division of computational workload between the CPU and GPU in the pure-CPU MoE fallback backend, and explain the core architectural trade-off of this design.

Learn Before

Related