Self-distillation has several practical applications in deep learning. It is used for data augmentation, where the self-knowledge of the augmentation process is distilled into the model. Additionally, it is utilized for sequentially optimizing deep neural network models that share the same architecture.

University of California, Berkeley

Google

Self-distillation is a special case of online distillation where the teacher and student use the same networks. Essentially, the student learns knowledge on its own.  


Self Distillation


- Knowledge from deeper sections of the network distilled into shallow sections.
- Self attention distillation method: Used for lane detection, the network uses attention maps of its own layers as dilation targets for its lower layers. 
- Snapshot distillation: A supervised training process whose knowledge in earlier epochs (teacher) is transferred its later epochs (student) 
- Self-knowledge distillation method: Matching predicted probabilities (feature representations of training model) to reflect data similarities in feature embedding space 

Learn Before

Related