- Knowledge from deeper sections of the network distilled into shallow sections.
- Self attention distillation method: Used for lane detection, the network uses attention maps of its own layers as dilation targets for its lower layers. 
- Snapshot distillation: A supervised training process whose knowledge in earlier epochs (teacher) is transferred its later epochs (student) 
- Self-knowledge distillation method: Matching predicted probabilities (feature representations of training model) to reflect data similarities in feature embedding space 

University of California, Berkeley

Self-distillation is a special case of online distillation where the teacher and student use the same networks. Essentially, the student learns knowledge on its own.  


Self Distillation


Self Distillation Methods 


- Data augmentation, where the self knowledge of augmentation is distilled into the model
- Optimizing deep models with same architecture one by one  

Learn Before

Related