Sparsity means that the activated experts should be sparse among all sub-networks for computational efficiency. This can be achieved by calculating a SoftMax score for each expert, and only activate the top few. 

Sparsity for MoE

Load balancing means that the work load should be balanced among all expert networks. There are various methods to address this problem, such as implementing auxiliary loss function, employing workload threshold, introducing local group dispatching, solving as linear assignment problem, etc.

Load Balancing for MoE

In MoE with learned routing, the router learns how to allocate the inputs to different expert network. To increase the computation efficiency, the computation of the router should not significantly slow the entire model. Furthermore, there are two key points to address: sparsity and load balancing.


New York University

Mixture of Expert (MoE) takes the advantage of increased amount of parameters without increasing the computation load. A layer in such a model consists of several expert network, and only part of them will be activated during inference. One key component of MoE model is the routing mechanism, which can be divided into two types: learned routing and unlearnable routing.

Mixture of Expert in Dynamic Neural Network

Xu, C., & McAuley, J. (2022). A Survey on Dynamic Neural Networks for Natural Language Processing. ArXiv, abs/2202.07101.

A Survey on Dynamic Neural Networks for Natural Language Processing

MoE with Learned Routing

Compared with MoE with learned router, unlearnable routing mechanism simplifies the computation by getting rid of load balancing process. Unlearnable routing can be achieved by hashing function, which hash a token to an expert, or by optimizing the consistency between a pair of randomly selected experts and allocating the tokens randomly.

Learn Before

Related

Learn After