Response based knowledge is simple, effective, and widely-applicable. It refers to the neural response of the teacher model’s last output layer. It copies the teacher model’s final prediction exactly. 


Response based knowledge


Feature-based knowledge builds on the concept of representation learning. It is especially suitable for training thinner and deeper networks. 

Feature-based knowledge



Relation-based knowledge investigates the relationships between different layers or data samples, rather than outputs of specific layers. 

Relation-based knowledge 


There are different types of knowledge in knowledge distillation (KD). They are 
- Response based knowledge
- Feature-based knowledge
- Relation-based knowledge  


University of California, Berkeley

- Knowledge
- Distillation algorithm
- Teacher-student architecture


Components of a Knowledge Distillation System


Gou, J., Yu, B., Maybank, S.J., & Tao, D. (2021). Knowledge Distillation: A Survey. Int. J. Comput. Vis., 129, 1789-1819.

Link: https://arxiv.org/pdf/2006.05525.pdf

Knowledge Distillation: A Survey

Categories of Knowledge 

There are 3 categories of distillation (training) schemes for teacher and student models. They are divided according to whether the teacher and the student model are updated simultaneously or not. 

- Offline Distillation 
- Online Distillation
- Self-Distillation 

Distillation (Training) Schemes 

Teacher-student architecture is a carrier to form the knowledge transfer. Focused on the relationship between the teacher and student model structure, it consists of selecting and designing proper structures of teacher and student. 


Teacher-Student Architecture



Directly matching the response-based knowledge, feature-based knowledge, or representation distributions in feature space between the the teacher and student models is a straightforward method for knowledge transfer. There are algorithms for different types of distillation methods for knowledge transfer. 

Learn Before

Related

Learn After