Similar activations of input pairs from teacher networks are transferred into the student network. Pairwise similarities are preserved. (Chen et al., 2021) (Passalis and Tefas, 2019; Passalis et al., 2020) (Tung and Mori, 2019)

University of California, Berkeley

Google


Individual knowledge distillation can be implemented by directly distilling the teacher’s individual soft targets into the student. This transferred knowledge contains feature information and mutual relations of data samples. 

Knowledge Distillation Methods 


The student network is learned by feature embedding, and relations between data samples are modeled as probabilistic distribution using feature representations of data. The teacher and student probability distributions are matched by knowledge transfer. (Peng et al. (2019)). 

Relational Knowledge Distillation


Distilled knowledge contains instance-level information and correlations between instances. Student network learns correlation between instances using corelation congruence for distillation. (Peng et al. (2019a)). 


Correlation Congruence Knowledge Distillation 


Similarity-Preserving Knowledge Distillation

In knowledge distillation, an instance relationship graph models transferred knowledge that contains instance features, instance relationships, and feature space transformations across layers.

Learn Before

Related