- Knowledge from deeper sections of the network distilled into shallow sections.
- Self attention distillation method: Used for lane detection, the network uses attention maps of its own layers as dilation targets for its lower layers. 
- Snapshot distillation: A supervised training process whose knowledge in earlier epochs (teacher) is transferred its later epochs (student) 
- Self-knowledge distillation method: Matching predicted probabilities (feature representations of training model) to reflect data similarities in feature embedding space 

Self Distillation Methods 


- Data augmentation, where the self knowledge of augmentation is distilled into the model
- Optimizing deep models with same architecture one by one  

Use Cases

Self-distillation is a special case of online distillation where the teacher and student use the same networks. Essentially, the student learns knowledge on its own.  


University of California, Berkeley

There are 3 categories of distillation (training) schemes for teacher and student models. They are divided according to whether the teacher and the student model are updated simultaneously or not. 

- Offline Distillation 
- Online Distillation
- Self-Distillation 

Distillation (Training) Schemes 

Gou, J., Yu, B., Maybank, S.J., & Tao, D. (2021). Knowledge Distillation: A Survey. Int. J. Comput. Vis., 129, 1789-1819.

Link: https://arxiv.org/pdf/2006.05525.pdf

Knowledge Distillation: A Survey

Offline methods assume the teacher knowledge/model is already defined, and focus on improving knowledge transfer, such as design of knowledge and loss functions for feature or distribution matching. They usually have one-way knowledge transfer and two-phase training procedure. 


Offline Distillation

In online distillation, the teacher and student models are updated simultaneously. It seeks to improve the student performance when a large-capacity high performance teacher model is not available. 


Learn Before

Related

Learn After