Concept icon
Concept

Pure-CPU MoE Fallback Backend

On consumer edge platforms where the full expert pool cannot be locked in pinned memory or registered for Direct Memory Access (DMA)—such as under restrictive operating systems or driver environments—the serving system provides a pure-CPU MoE fallback backend. Under this fallback mode, expert parameters reside in standard pageable host RAM, and all routed expert computations are performed entirely on the CPU. Non-expert model layers remain resident on the GPU, so that only activation-sized tensor inputs, routing metadata, and aggregated expert outputs are transferred across the CPU–GPU bus. While trading off peak PCIe transfer bandwidth, this execution strategy ensures universal deployability on systems incapable of supporting pinned DMA transfers.

0

1

Concept icon
Updated 2026-09-07

Tags

Prep Sessions

Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor

Ch.4 Platform Adaptation and Performance Evaluation - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor

Expert Storage Formats and Platform Adaptation - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor