The distillation loss of relation-based knowledge, based on the relations of feature maps is

• $L_{RelD}(f_t, f_s) = L_{R^1}(\psi_t(\acute{f_t}, \check{f_t}), \psi_s(\acute{f_s}, \check{f_s}))$

• $f_t, f_s$ are feature maps of teacher and student models

• $\acute{f_t}, \check{f_t}$ are pairs of feature maps chosen from the teacher

• $\acute{f_s}, \check{f_s}$ are pairs of feature maps chosen from the student

• $\psi_t(.), \psi_s(.)$ are similarity functions for Pais of feature maps from the models

• $L{R^1}(.)$ is the correlation function between teacher and student feature maps