The distillation loss for relation-based knowledge transfer, based on the relations of feature maps, is calculated as:

$$L_{RelD}(f_t, f_s) = L_{R^1}(\psi_t(\acute{f_t}, \check{f_t}), \psi_s(\acute{f_s}, \check{f_s}))$$

Where:
- $$f_t$$ and $$f_s$$ are feature maps of the teacher and student models, respectively.
- $$\acute{f_t}$$ and $$\check{f_t}$$ are pairs of feature maps chosen from the teacher.
- $$\acute{f_s}$$ and $$\check{f_s}$$ are pairs of feature maps chosen from the student.
- $$\psi_t(\cdot)$$ and $$\psi_s(\cdot)$$ are similarity functions for pairs of feature maps from the models.
- $$L_{R^1}(\cdot)$$ is the correlation function between the teacher and student feature maps.

Distillation Loss for Relation-Based Knowledge

In relation-based knowledge transfer, feature map relationships between two neural network layers can be modeled using a Gram matrix. This matrix summarizes relations by calculating the inner products between pairs of feature maps from the two layers. Singular Value Decomposition (SVD) is then applied to the resulting Gram matrix to extract essential information from these feature map correlations.

Gram Matrix for Feature Map Relationships

Relation-based knowledge distillation methods train a student network to preserve relationships represented by a teacher network, such as pairwise similarities, correlations between instances, probabilistic relations among samples, or instance relationship graphs.

Relation-Based Knowledge Distillation Methods

Relation-based knowledge in knowledge distillation consists of relationships represented by a teacher model—such as pairwise similarities, feature-map correlations, probabilistic relations among samples, or instance relationship graphs—that a student model learns to preserve. It focuses on relations among representations or samples rather than isolated layer outputs.

University of California, Berkeley

Google

There are different types of knowledge in knowledge distillation (KD). They are 
- Response based knowledge
- Feature-based knowledge
- Relation-based knowledge  


Categories of Knowledge 

Feature-based knowledge is a knowledge distillation approach that uses the feature maps produced by a teacher model's intermediate layers, not just its final output, as knowledge to transfer to a student model. It builds on the concept of representation learning, since deep networks learn multiple levels of increasingly abstracted feature representations, and it is especially suitable for training thinner and deeper student networks.

Feature-based knowledge

Response-based knowledge in knowledge distillation consists of a teacher model's final-layer outputs—such as logits, class probabilities, bounding-box offsets, or landmark heat maps—that guide a student model. The student is trained to reduce a divergence between its responses and the teacher's; it does not necessarily copy the teacher's final prediction exactly.

Learn Before

Related

Learn After