Deep neural networks are good at representation learning, or learning multiple levels of increasingly abstracted feature representation. This means the output of the last layer and the intermediate layers (feature maps) can be used as knowledge. 

University of California, Berkeley

Feature-based knowledge builds on the concept of representation learning. It is especially suitable for training thinner and deeper networks. 

Feature-based knowledge


Representation Learning


Intermediate representations tries to match the feature activations of the teacher and student. This was introduced to provide hints for training the student. 

Idea


Distillation loss for feature-based learning transfer is 

$L_{FeaD}(f_t(x), f_s(x)) = L_F(\phi_t(f_t(x)), \phi_s(f_s(x))) $

-$f_t(x)$ and $f_s(x)$ are the feature maps of the intermediate layers of the teacher and student models 
-$\phi_s(f_t(x))$ and $\phi_s(f_s(x))$ are applied when feature maps of the two models have different shape 
-$L_f(.)$ is the similarity function for matching the models’ feature maps 

Learn Before

Related