Cascading is a special type of early exit at model level instead of layer level. For example, CascadeBERT executes a sequence of models one by one, and it exits when one of them outputs a score that reaches the threshold.

New York University

Early exit method aims to terminate the computation in earlier layer to reduce computation and achieve a better performance. The key of such method is the exit criterion, which can be categorized as: confidence-based, ensemble-based and learning-based.

Early Exit

Xu, C., & McAuley, J. (2022). A Survey on Dynamic Neural Networks for Natural Language Processing. ArXiv, abs/2202.07101.

A Survey on Dynamic Neural Networks for Natural Language Processing

The benefits of early exit model include better performance, higher computational speed, and adversarial robustness. In addition, the speed-accuracy trade-off can be adjusted by tuning the exit threshold during inference without retraining the model.
The drawback comes along is that the batch size is usually 1 since early exist is applied per-instance. 


Benefit and Drawback of Early Exit

In confidence-based early exit model, a indicator of confidence of the prediction is defined, and when this indicator hits the threshold, the model will exits. 

Confidence-Based Early Exit

One draw back of confidence-based early exit is wasted computation when the confidence indicator does not reach the threshold. On the contrary, ensemble-based early exit model makes use of predictions of multiple internal classifier and makes better prediction. 

Ensemble-Based Early Exit

In this genre of model, a classifier is directly trained to predict whether the current internal classifier has made the correct prediction. 

Learn Before

Related