Skipping and early stopping increase the computational efficiency for long input sequences by skipping some tokens or by stopping reading. With various implementation methods, it is possible to skip one or multiple time steps, to jump back to previous tokens, to jump to next structural tokens, and to stop reading directly. Intuitively, these techniques simulate how human read and extract information from texts.

New York University

Skimming is widely used in Recurrent Neural Network (RNN), where inputs are sequentially processed in time steps. Skimming models save computation timewisely by dynamically allocating computations to different time steps. It intuitively matches how human read and extract information: focusing on the important part and ignoring parts with little importance.
There are three types of skimming:
- Skipping and early stopping
- Computation reduction
- Dynamic hierarchical RNN

Skimming

Xu, C., & McAuley, J. (2022). A Survey on Dynamic Neural Networks for Natural Language Processing. ArXiv, abs/2202.07101.

A Survey on Dynamic Neural Networks for Natural Language Processing

Skipping and Early Stopping

Computation reduction method reduces computation for certain time steps instead of completely skipping these steps. If computation reduction is activated at a given time step, based on different implementation, the hidden states will be updated by a masked weight matrix, or some part of the hidden states will be updated by another smaller network. In both cases, computations are reduced.

Computation Reduction in Dynamic Neural Network

Dynamical hierarchical RNN will adaptably increase the computation by calling an upper layer RNN when necessary. 

Learn Before

Related