Deep neural networks are good at representation learning, or learning multiple levels of increasingly abstracted feature representation. This means the output of the last layer and the intermediate layers (feature maps) can be used as knowledge. 

Representation Learning


Intermediate representations tries to match the feature activations of the teacher and student. This was introduced to provide hints for training the student. 

Idea


Distillation loss for feature-based learning transfer is 

$L_{FeaD}(f_t(x), f_s(x)) = L_F(\phi_t(f_t(x)), \phi_s(f_s(x))) $

-$f_t(x)$ and $f_s(x)$ are the feature maps of the intermediate layers of the teacher and student models 
-$\phi_s(f_t(x))$ and $\phi_s(f_s(x))$ are applied when feature maps of the two models have different shape 
-$L_f(.)$ is the similarity function for matching the models’ feature maps 

Distillation Loss


Feature-based knowledge builds on the concept of representation learning. It is especially suitable for training thinner and deeper networks. 

University of California, Berkeley

There are different types of knowledge in knowledge distillation (KD). They are 
- Response based knowledge
- Feature-based knowledge
- Relation-based knowledge  


Categories of Knowledge 

Response based knowledge is simple, effective, and widely-applicable. It refers to the neural response of the teacher model’s last output layer. It copies the teacher model’s final prediction exactly. 


Response based knowledge


Feature-based knowledge



Relation-based knowledge investigates the relationships between different layers or data samples, rather than outputs of specific layers. 

Learn Before

Related

Learn After