The benefits of early exit model include better performance, higher computational speed, and adversarial robustness. In addition, the speed-accuracy trade-off can be adjusted by tuning the exit threshold during inference without retraining the model.
The drawback comes along is that the batch size is usually 1 since early exist is applied per-instance. 


Benefit and Drawback of Early Exit

In confidence-based early exit model, a indicator of confidence of the prediction is defined, and when this indicator hits the threshold, the model will exits. 

Confidence-Based Early Exit

One draw back of confidence-based early exit is wasted computation when the confidence indicator does not reach the threshold. On the contrary, ensemble-based early exit model makes use of predictions of multiple internal classifier and makes better prediction. 

Ensemble-Based Early Exit

In this genre of model, a classifier is directly trained to predict whether the current internal classifier has made the correct prediction. 

Learning-Based Early Exit

Cascading is a special type of early exit at model level instead of layer level. For example, CascadeBERT executes a sequence of models one by one, and it exits when one of them outputs a score that reaches the threshold.

Cascading as Early Exit

Early exit method aims to terminate the computation in earlier layer to reduce computation and achieve a better performance. The key of such method is the exit criterion, which can be categorized as: confidence-based, ensemble-based and learning-based.

New York University

There are three types of techniques to achieve dynamic neural network: Skimming, MoE, and Early Exit. The common idea is to adjust computation based on inputs.
- Skimming
    - Skipping and Early Stopping
    - Computation Reduction
    - Dynamic Hierarchical RNN
- Mixture of Experts
    - Learned Routing
    - Unlearnable Routing
- Early Exit
    - Confidence-based
    - Ensemble-based
    - Learning-based
    - Cascading

Types of Dynamic Neural Networks Adopted in NLP

Xu, C., & McAuley, J. (2022). A Survey on Dynamic Neural Networks for Natural Language Processing. ArXiv, abs/2202.07101.

A Survey on Dynamic Neural Networks for Natural Language Processing

Skimming is widely used in Recurrent Neural Network (RNN), where inputs are sequentially processed in time steps. Skimming models save computation timewisely by dynamically allocating computations to different time steps. It intuitively matches how human read and extract information: focusing on the important part and ignoring parts with little importance.
There are three types of skimming:
- Skipping and early stopping
- Computation reduction
- Dynamic hierarchical RNN

Skimming

Mixture of Expert (MoE) takes the advantage of increased amount of parameters without increasing the computation load. A layer in such a model consists of several expert network, and only part of them will be activated during inference. One key component of MoE model is the routing mechanism, which can be divided into two types: learned routing and unlearnable routing.

Learn Before

Related

Learn After