This transferred knowledge contains instance features, instance relationships, and feature space transformation cross layers. 

Instance Relationship Graph 


The student network is learned by feature embedding, and relations between data samples are modeled as probabilistic distribution using feature representations of data. The teacher and student probability distributions are matched by knowledge transfer. (Peng et al. (2019)). 

Relational Knowledge Distillation


Similar activations of input pairs from teacher networks is transferred into the student network. Pairwise similarities are preserved. (Chen et al., 2021) (Passalis and Tefas, 2019; Pessalis et al., 20201) (Tung and More, 2019)

Similarity-Preserving Knowledge Distillation


Distilled knowledge contains instance-level information and correlations between instances. Student network learns correlation between instances using corelation congruence for distillation. (Peng et al. (2019a)). 


Correlation Congruence Knowledge Distillation 



Individual knowledge distillation can be implemented by directly distilling the teacher’s individual soft targets into the student. This transferred knowledge contains feature information and mutual relations of data samples. 

University of California, Berkeley


Relation-based knowledge investigates the relationships between different layers or data samples, rather than outputs of specific layers. 

Relation-based knowledge 


Flow of solution process: Defined by the Gram matrix between 2 layers, it summarizes relations between pairs of feature maps and is calculated using the inner products between features from 2 layers. SVD is used to extract information from correlations between feature maps.

Exploring Feature Map Relationships


The distillation loss of relation-based knowledge, based on the relations of feature maps is
- $L_{RelD}(f_t, f_s) = L_{R^1}(\psi_t(\acute{f_t}, \check{f_t}), \psi_s(\acute{f_s}, \check{f_s})) $

- $f_t, f_s$ are feature maps of teacher and student models
- $\acute{f_t}, \check{f_t}$ are pairs of feature maps chosen from the teacher 
- $\acute{f_s}, \check{f_s}$ are pairs of feature maps chosen from the student 
- $\psi_t(.), \psi_s(.)$ are similarity functions for Pais of feature maps from the models 
- $L{R^1}(.)$ is the correlation function between teacher and student feature maps

Learn Before

Related

Learn After