The distillation loss for feature-based knowledge transfer matches the feature maps of intermediate layers between teacher and student models. It is calculated as:

$L_{FeaD}(f_t(x), f_s(x)) = L_F(\phi_t(f_t(x)), \phi_s(f_s(x)))$

Where:

• $f_t(x)$ and $f_s(x)$ are the feature maps of the intermediate layers of the teacher and student models, respectively.
• $\phi_t(\cdot)$ and $\phi_s(\cdot)$ are transformation functions applied when the feature maps of the two models have different shapes.
• $L_F(\cdot)$ is the similarity function used for matching the models' feature maps.