The student network is learned by feature embedding, and relations between data samples are modeled as probabilistic distribution using feature representations of data. The teacher and student probability distributions are matched by knowledge transfer. (Peng et al. (2019)). 

Relational Knowledge Distillation


Distilled knowledge contains instance-level information and correlations between instances. Student network learns correlation between instances using corelation congruence for distillation. (Peng et al. (2019a)). 


Correlation Congruence Knowledge Distillation 


Similar activations of input pairs from teacher networks are transferred into the student network. Pairwise similarities are preserved. (Chen et al., 2021) (Passalis and Tefas, 2019; Passalis et al., 2020) (Tung and Mori, 2019)

Similarity-Preserving Knowledge Distillation

In knowledge distillation, an instance relationship graph models transferred knowledge that contains instance features, instance relationships, and feature space transformations across layers.

Instance Relationship Graph

Relation-based knowledge distillation methods train a student network to preserve relationships represented by a teacher network, such as pairwise similarities, correlations between instances, probabilistic relations among samples, or instance relationship graphs.

University of California, Berkeley

Google


Relation-based knowledge investigates the relationships between different layers or data samples, rather than outputs of specific layers. 

Relation-based knowledge 


The distillation loss for relation-based knowledge transfer, based on the relations of feature maps, is calculated as:

$$L_{RelD}(f_t, f_s) = L_{R^1}(\psi_t(\acute{f_t}, \check{f_t}), \psi_s(\acute{f_s}, \check{f_s}))$$

Where:
- $$f_t$$ and $$f_s$$ are feature maps of the teacher and student models, respectively.
- $$\acute{f_t}$$ and $$\check{f_t}$$ are pairs of feature maps chosen from the teacher.
- $$\acute{f_s}$$ and $$\check{f_s}$$ are pairs of feature maps chosen from the student.
- $$\psi_t(\cdot)$$ and $$\psi_s(\cdot)$$ are similarity functions for pairs of feature maps from the models.
- $$L_{R^1}(\cdot)$$ is the correlation function between the teacher and student feature maps.

Distillation Loss for Relation-Based Knowledge

In relation-based knowledge transfer, feature map relationships between two neural network layers can be modeled using a Gram matrix. This matrix summarizes relations by calculating the inner products between pairs of feature maps from the two layers. Singular Value Decomposition (SVD) is then applied to the resulting Gram matrix to extract essential information from these feature map correlations.

Learn Before

Related

Learn After