Skimming is widely used in Recurrent Neural Network (RNN), where inputs are sequentially processed in time steps. Skimming models save computation timewisely by dynamically allocating computations to different time steps. It intuitively matches how human read and extract information: focusing on the important part and ignoring parts with little importance.
There are three types of skimming:
- Skipping and early stopping
- Computation reduction
- Dynamic hierarchical RNN

Skimming

Mixture of Expert (MoE) takes the advantage of increased amount of parameters without increasing the computation load. A layer in such a model consists of several expert network, and only part of them will be activated during inference. One key component of MoE model is the routing mechanism, which can be divided into two types: learned routing and unlearnable routing.

Mixture of Expert in Dynamic Neural Network

Early exit method aims to terminate the computation in earlier layer to reduce computation and achieve a better performance. The key of such method is the exit criterion, which can be categorized as: confidence-based, ensemble-based and learning-based.

Early Exit

There are three types of techniques to achieve dynamic neural network: Skimming, MoE, and Early Exit. The common idea is to adjust computation based on inputs.
- Skimming
    - Skipping and Early Stopping
    - Computation Reduction
    - Dynamic Hierarchical RNN
- Mixture of Experts
    - Learned Routing
    - Unlearnable Routing
- Early Exit
    - Confidence-based
    - Ensemble-based
    - Learning-based
    - Cascading

New York University

Increasing model capacity is an effective approach for better performance in NLP. However, larger model capacity will increase computational load and memory consumption, which becomes a bottleneck for increasing model capacity. Dynamic Neural Network is used to overcome this issue, by adjusting the computational path based on the inputs.

Dynamic Neural Network in Natural Language Processing

Han, Y., Huang, G., Song, S., Yang, L., Wang, H., & Wang, Y. (2021). Dynamic Neural Networks: A Survey. IEEE transactions on pattern analysis and machine intelligence, PP.
https://arxiv.org/abs/2102.04906

Dynamic Neural Networks: A Survey

Types of Dynamic Neural Networks Adopted in NLP

Xu, C., & McAuley, J. (2022). A Survey on Dynamic Neural Networks for Natural Language Processing. ArXiv, abs/2202.07101.

A Survey on Dynamic Neural Networks for Natural Language Processing

The challenge and future direction for dynamic neural network includes:
- Evaluation: evaluation of different DNNs is difficult since it is hard to define break-point at the exact same computation cost for different methods.
- Data Parallelism: some DNN methods like skipping requires a batch size of 1, which is inefficient for batched inference.
- Optimized Runtime: most hardware and libraries are not well-optimized for DNN models.
- Theoretical Analysis: DNNs are not solidly backed by theoretical analysis.
- Explainability: The decision-making process of the DNNs can give insights to explain the model prediction.

Learn Before

Related

Learn After