Teacher and student model setup are almost pre-fixed in size and structure during distillation, causing a model capacity gap. It is missing a way to design the architectures and an explanation for why their architectures are determined by these model setups. 


Recent work 


Model capacity gap between the two models can degrade knowledge transfer. Methods for controlling the reduction of model complexity include techniques for mitigating the training gap. 


Knowledge Transfer Methods 


Teacher-student architecture is a carrier to form the knowledge transfer. Focused on the relationship between the teacher and student model structure, it consists of selecting and designing proper structures of teacher and student. 


University of California, Berkeley

- Knowledge
- Distillation algorithm
- Teacher-student architecture


Components of a Knowledge Distillation System


There are different types of knowledge in knowledge distillation (KD). They are 
- Response based knowledge
- Feature-based knowledge
- Relation-based knowledge  


Categories of Knowledge 

There are 3 categories of distillation (training) schemes for teacher and student models. They are divided according to whether the teacher and the student model are updated simultaneously or not. 

- Offline Distillation 
- Online Distillation
- Self-Distillation 

Distillation (Training) Schemes 

Teacher-Student Architecture



Directly matching the response-based knowledge, feature-based knowledge, or representation distributions in feature space between the the teacher and student models is a straightforward method for knowledge transfer. There are algorithms for different types of distillation methods for knowledge transfer. 

Learn Before

Related

Learn After