Learn Before
Edge MoE Serving and Architectural Bottlenecks - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Prefill Transfer and Context Recomputation Challenges - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Decode Cache Misses and Host Bandwidth Bottlenecks - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Non-Dedicated Edge Resource Dynamics - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Mixture-of-Experts (MoE) for Efficient Inference
Systems Challenge of Edge MoE Serving
Mixture-of-Experts (MoE) architectures make frontier-scale model execution feasible on edge devices because routing each token to only a sparse subset of experts () drastically reduces the active parameter footprint and per-token computation. For example, a model may activate only a small fraction of its total parameter count for any single token, allowing the active compute footprint to fit within consumer GPU memory. However, architectural sparsity reduces computation without proportionally shrinking the memory required to store the full expert pool. When the complete model parameter footprint greatly exceeds GPU memory, inactive experts must reside in CPU host memory or secondary storage and enter the execution path on demand. This discrepancy creates the central systems challenge of edge MoE serving: sparse activation makes on-device computation feasible, but storing and streaming the complete expert pool makes efficient serving difficult.
0
1
Tags
Prep Sessions
Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Ch.1 Edge Serving Bottlenecks and Dynamics - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Edge MoE Serving and Architectural Bottlenecks - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Prefill Transfer and Context Recomputation Challenges - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Decode Cache Misses and Host Bandwidth Bottlenecks - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Non-Dedicated Edge Resource Dynamics - Edge-Native Mixture-of-Experts Serving with FreeToken @ University of Michigan - Ann Arbor
Related
Systems Challenge of Edge MoE Serving
Mixture-of-Experts (MoE) for Efficient Inference
Expert Transfer Bottleneck in Edge MoE Prefill
Context Recomputation and Checkpoint Invalidation in Agentic Serving
Systems Challenge of Edge MoE Serving
Prefilling as a Compute-Bound Process
Static Expert Placement Inefficiency in Edge MoE Decode
Host DRAM Bandwidth Bottleneck in CPU Expert Execution
Hardware-Specific Trade-Off in Serving MoE Decode Cache Misses
Systems Challenge of Edge MoE Serving
Dynamic VRAM Availability and Memory Split Shifts in Edge MoE Serving
Recurring Engine Bootstrap Bottleneck in Edge MoE Serving
Systems Challenge of Edge MoE Serving
Experts as Modular FFNs in LLM MoE Models
A large language model is deployed for inference across 8 powerful processing units. In one configuration, the entire model's computational graph is activated across all 8 units for every input. In a second configuration, the model is structured with 8 distinct 'expert' sub-networks, one on each unit. For a given input, a routing mechanism selects only the 2 most relevant expert sub-networks to perform computations. What is the primary efficiency benefit of the second configuration for processin
In a Mixture-of-Experts (MoE) architecture, all expert sub-networks must be hosted on a single hardware device.
Systems Challenge of Edge MoE Serving
What effect does selective execution have on computational efficiency and model quality in MoE inference?
Based on MoE operational principles, which expert sub-networks are activated for computation on this request?
Learn After
How does a Mixture-of-Experts (MoE) architecture allow the computation of a frontier-scale model to fit within consumer GPU memory on an edge device?
What constitutes the central systems challenge of serving Mixture-of-Experts (MoE) models on edge devices?
What is the impact of architectural sparsity on the memory required to store an MoE model's full expert pool?
Where do inactive experts reside when the complete parameter footprint of an MoE model exceeds the memory of an edge device's GPU?
Match each MoE serving parameter or edge concept to its corresponding description.
Identify the primary systems bottleneck causing latency degradation in this scenario, and explain why expanding the total expert pool () while keeping the number of active experts () constant worsens this bottleneck.
In an edge MoE deployment where the full model footprint exceeds GPU capacity, by what mechanism do inactive experts participate in token processing?
In edge Mixture-of-Experts (MoE) serving, consumer GPU memory only needs to accommodate the active compute footprint for a given token rather than the entire parameter footprint simultaneously during execution.
Which two operational metrics are drastically reduced when an MoE architecture routes each token to a sparse subset of experts ()?