Early Exit in DeeBERT is achieved as follows:

• Training stage:
  1. Train and fine tune a BERT on downstream tasks.
  2. Freeze the parameters of BERT, insert a linear classifier after each Transformer layer.
  3. Train the classifier by minimizing the sum of their cross-entropy loss.
• Inference stage: early exits when an internal classifier output a distribution that has entropy lower than the predefined threshold.