Skipping and early stopping increase the computational efficiency for long input sequences by skipping some tokens or by stopping reading. With various implementation methods, it is possible to skip one or multiple time steps, to jump back to previous tokens, to jump to next structural tokens, and to stop reading directly. Intuitively, these techniques simulate how human read and extract information from texts.

Skipping and Early Stopping

Computation reduction method reduces computation for certain time steps instead of completely skipping these steps. If computation reduction is activated at a given time step, based on different implementation, the hidden states will be updated by a masked weight matrix, or some part of the hidden states will be updated by another smaller network. In both cases, computations are reduced.

Computation Reduction in Dynamic Neural Network

Dynamical hierarchical RNN will adaptably increase the computation by calling an upper layer RNN when necessary. 

Dynamical Hierarchical RNN

Skimming is widely used in Recurrent Neural Network (RNN), where inputs are sequentially processed in time steps. Skimming models save computation timewisely by dynamically allocating computations to different time steps. It intuitively matches how human read and extract information: focusing on the important part and ignoring parts with little importance.
There are three types of skimming:
- Skipping and early stopping
- Computation reduction
- Dynamic hierarchical RNN

New York University

There are three types of techniques to achieve dynamic neural network: Skimming, MoE, and Early Exit. The common idea is to adjust computation based on inputs.
- Skimming
    - Skipping and Early Stopping
    - Computation Reduction
    - Dynamic Hierarchical RNN
- Mixture of Experts
    - Learned Routing
    - Unlearnable Routing
- Early Exit
    - Confidence-based
    - Ensemble-based
    - Learning-based
    - Cascading

Types of Dynamic Neural Networks Adopted in NLP

Xu, C., & McAuley, J. (2022). A Survey on Dynamic Neural Networks for Natural Language Processing. ArXiv, abs/2202.07101.

A Survey on Dynamic Neural Networks for Natural Language Processing

Skimming

Mixture of Expert (MoE) takes the advantage of increased amount of parameters without increasing the computation load. A layer in such a model consists of several expert network, and only part of them will be activated during inference. One key component of MoE model is the routing mechanism, which can be divided into two types: learned routing and unlearnable routing.

Mixture of Expert in Dynamic Neural Network

Early exit method aims to terminate the computation in earlier layer to reduce computation and achieve a better performance. The key of such method is the exit criterion, which can be categorized as: confidence-based, ensemble-based and learning-based.

Learn Before

Related

Learn After