Flow of solution process: Defined by the Gram matrix between 2 layers, it summarizes relations between pairs of feature maps and is calculated using the inner products between features from 2 layers. SVD is used to extract information from correlations between feature maps.

Exploring Feature Map Relationships


The distillation loss of relation-based knowledge, based on the relations of feature maps is
- $L_{RelD}(f_t, f_s) = L_{R^1}(\psi_t(\acute{f_t}, \check{f_t}), \psi_s(\acute{f_s}, \check{f_s})) $

- $f_t, f_s$ are feature maps of teacher and student models
- $\acute{f_t}, \check{f_t}$ are pairs of feature maps chosen from the teacher 
- $\acute{f_s}, \check{f_s}$ are pairs of feature maps chosen from the student 
- $\psi_t(.), \psi_s(.)$ are similarity functions for Pais of feature maps from the models 
- $L{R^1}(.)$ is the correlation function between teacher and student feature maps

Distillation Loss 



Individual knowledge distillation can be implemented by directly distilling the teacher’s individual soft targets into the student. This transferred knowledge contains feature information and mutual relations of data samples. 

Knowledge Distillation Methods 



Relation-based knowledge investigates the relationships between different layers or data samples, rather than outputs of specific layers. 

University of California, Berkeley

There are different types of knowledge in knowledge distillation (KD). They are 
- Response based knowledge
- Feature-based knowledge
- Relation-based knowledge  


Categories of Knowledge 

Response based knowledge is simple, effective, and widely-applicable. It refers to the neural response of the teacher model’s last output layer. It copies the teacher model’s final prediction exactly. 


Response based knowledge


Feature-based knowledge builds on the concept of representation learning. It is especially suitable for training thinner and deeper networks. 

Learn Before

Related

Learn After