Model capacity gap between the two models can degrade knowledge transfer. Methods for controlling the reduction of model complexity include techniques for mitigating the training gap. 


Knowledge Transfer Methods 


In traditional knowledge distillation, the teacher and student model architectures are almost always pre-fixed in size and structure. This fixed configuration often results in a model capacity gap, where the smaller student cannot effectively absorb the larger teacher's knowledge. Furthermore, standard distillation frameworks lack a systematic methodology to design these architectures and do not explain how the pre-fixed model setups constrain the architectural choices.

Limitations of Pre-fixed Teacher-Student Architectures in Knowledge Distillation

In knowledge distillation, the teacher-student architecture is the pair of network structures that serves as the carrier for knowledge transfer. Designing it involves selecting suitable structures for the teacher model and the student model, because the capacity gap between the two models affects how effectively the student can absorb the teacher's knowledge.

University of California, Berkeley

Google

- Knowledge
- Distillation algorithm
- Teacher-student architecture


Components of a Knowledge Distillation System


There are different types of knowledge in knowledge distillation (KD). They are 
- Response based knowledge
- Feature-based knowledge
- Relation-based knowledge  


Categories of Knowledge 

There are 3 categories of distillation (training) schemes for teacher and student models. They are divided according to whether the teacher and the student model are updated simultaneously or not. 

- Offline Distillation 
- Online Distillation
- Self-Distillation 

Distillation (Training) Schemes 

Directly matching the response-based knowledge, feature-based knowledge, or relation-based knowledge between the teacher and student models is a common way to transfer knowledge in knowledge distillation (KD). Different algorithms specialize in different mechanisms for this transfer, including: - Adversarial Distillation - Multi-Teacher Distillation - Cross-Model Distillation - Graph-Based Distillation - Attention-Based Distillation - NAS-Based Distillation - Quantized Distillation - Lifelong Distillation - Data-Free Distillation

Learn Before

Related

Learn After