Self-distillation has several practical applications in deep learning. It is used for data augmentation, where the self-knowledge of the augmentation process is distilled into the model. Additionally, it is utilized for sequentially optimizing deep neural network models that share the same architecture.

Self-Distillation Use Cases

Self-distillation methods transfer supervision within a network or across stages of its training process:

- Deep-to-shallow distillation transfers knowledge from deeper network sections to shallower sections.
- Self-attention distillation uses attention maps from later layers as distillation targets for earlier layers; the supplied context identifies lane detection as an application.
- Snapshot distillation uses an earlier training snapshot as the teacher for a later snapshot.
- Self-knowledge distillation matches predicted probabilities so that they reflect similarities in the model's feature-embedding space.

Self-Distillation Methods

Self-distillation is a special case of online distillation where the teacher and student use the same networks. Essentially, the student learns knowledge on its own.  


University of California, Berkeley

There are 3 categories of distillation (training) schemes for teacher and student models. They are divided according to whether the teacher and the student model are updated simultaneously or not. 

- Offline Distillation 
- Online Distillation
- Self-Distillation 

Distillation (Training) Schemes 

Gou, J., Yu, B., Maybank, S.J., & Tao, D. (2021). Knowledge Distillation: A Survey. Int. J. Comput. Vis., 129, 1789-1819.

Link: https://arxiv.org/pdf/2006.05525.pdf

Knowledge Distillation: A Survey

Offline methods assume the teacher knowledge/model is already defined, and focus on improving knowledge transfer, such as design of knowledge and loss functions for feature or distribution matching. They usually have one-way knowledge transfer and two-phase training procedure. 


Offline Distillation

In online distillation, the teacher and student models are updated simultaneously. It seeks to improve the student performance when a large-capacity high performance teacher model is not available. 


Learn Before

Related

Learn After