The challenge and future direction for dynamic neural network includes:
- Evaluation: evaluation of different DNNs is difficult since it is hard to define break-point at the exact same computation cost for different methods.
- Data Parallelism: some DNN methods like skipping requires a batch size of 1, which is inefficient for batched inference.
- Optimized Runtime: most hardware and libraries are not well-optimized for DNN models.
- Theoretical Analysis: DNNs are not solidly backed by theoretical analysis.
- Explainability: The decision-making process of the DNNs can give insights to explain the model prediction.

New York University

Increasing model capacity is an effective approach for better performance in NLP. However, larger model capacity will increase computational load and memory consumption, which becomes a bottleneck for increasing model capacity. Dynamic Neural Network is used to overcome this issue, by adjusting the computational path based on the inputs.

Dynamic Neural Network in Natural Language Processing

Xu, C., & McAuley, J. (2022). A Survey on Dynamic Neural Networks for Natural Language Processing. ArXiv, abs/2202.07101.

A Survey on Dynamic Neural Networks for Natural Language Processing

There are three types of techniques to achieve dynamic neural network: Skimming, MoE, and Early Exit. The common idea is to adjust computation based on inputs.
- Skimming
    - Skipping and Early Stopping
    - Computation Reduction
    - Dynamic Hierarchical RNN
- Mixture of Experts
    - Learned Routing
    - Unlearnable Routing
- Early Exit
    - Confidence-based
    - Ensemble-based
    - Learning-based
    - Cascading

Learn Before

Related