Xu, C., & McAuley, J. (2022). A Survey on Dynamic Neural Networks for Natural Language Processing. ArXiv, abs/2202.07101. https://arxiv.org/abs/2202.07101

New York University

Google

Increasing model capacity is an effective approach for better performance in NLP. However, larger model capacity will increase computational load and memory consumption, which becomes a bottleneck for increasing model capacity. Dynamic Neural Network is used to overcome this issue, by adjusting the computational path based on the inputs.

Types of Dynamic Neural Networks Adopted in NLP

A Survey on Dynamic Neural Networks for Natural Language Processing

The challenge and future direction for dynamic neural network includes:
- Evaluation: evaluation of different DNNs is difficult since it is hard to define break-point at the exact same computation cost for different methods.
- Data Parallelism: some DNN methods like skipping requires a batch size of 1, which is inefficient for batched inference.
- Optimized Runtime: most hardware and libraries are not well-optimized for DNN models.
- Theoretical Analysis: DNNs are not solidly backed by theoretical analysis.
- Explainability: The decision-making process of the DNNs can give insights to explain the model prediction.

Learn Before

Related