Sparsity means that the activated experts should be sparse among all sub-networks for computational efficiency. This can be achieved by calculating a SoftMax score for each expert, and only activate the top few. 

New York University

In MoE with learned routing, the router learns how to allocate the inputs to different expert network. To increase the computation efficiency, the computation of the router should not significantly slow the entire model. Furthermore, there are two key points to address: sparsity and load balancing.


MoE with Learned Routing

Xu, C., & McAuley, J. (2022). A Survey on Dynamic Neural Networks for Natural Language Processing. ArXiv, abs/2202.07101.

A Survey on Dynamic Neural Networks for Natural Language Processing

Sparsity for MoE

Load balancing means that the work load should be balanced among all expert networks. There are various methods to address this problem, such as implementing auxiliary loss function, employing workload threshold, introducing local group dispatching, solving as linear assignment problem, etc.

Learn Before

Related