Offline methods assume the teacher knowledge/model is already defined, and focus on improving knowledge transfer, such as design of knowledge and loss functions for feature or distribution matching. They usually have one-way knowledge transfer and two-phase training procedure. 


Offline Distillation

In online distillation, the teacher and student models are updated simultaneously. It seeks to improve the student performance when a large-capacity high performance teacher model is not available. 


Online Distillation


Self-distillation is a special case of online distillation where the teacher and student use the same networks. Essentially, the student learns knowledge on its own.  


Self Distillation


There are 3 categories of distillation (training) schemes for teacher and student models. They are divided according to whether the teacher and the student model are updated simultaneously or not. 

- Offline Distillation 
- Online Distillation
- Self-Distillation 

University of California, Berkeley

- Knowledge
- Distillation algorithm
- Teacher-student architecture


Components of a Knowledge Distillation System


Gou, J., Yu, B., Maybank, S.J., & Tao, D. (2021). Knowledge Distillation: A Survey. Int. J. Comput. Vis., 129, 1789-1819.

Link: https://arxiv.org/pdf/2006.05525.pdf

Knowledge Distillation: A Survey

There are different types of knowledge in knowledge distillation (KD). They are 
- Response based knowledge
- Feature-based knowledge
- Relation-based knowledge  


Categories of Knowledge 

Distillation (Training) Schemes 

Teacher-student architecture is a carrier to form the knowledge transfer. Focused on the relationship between the teacher and student model structure, it consists of selecting and designing proper structures of teacher and student. 


Teacher-Student Architecture



Directly matching the response-based knowledge, feature-based knowledge, or representation distributions in feature space between the the teacher and student models is a straightforward method for knowledge transfer. There are algorithms for different types of distillation methods for knowledge transfer. 

Learn Before

Related

Learn After