- Add a teacher assistant mitigates the training gap, which is further reduced by residual learning, where the assistant structure is used to learn the residual error. 
- Minimize the difference in teacher and student model structure by combining network quantization with knowledge distillation.
- Structure compression method: Transfers the knowledge of multiple layers to a single layer. In online settings, teacher networks are ensembles of similarly-structured student networks. 

Techniques for Mitigating the Training Gap 


Model capacity gap between the two models can degrade knowledge transfer. Methods for controlling the reduction of model complexity include techniques for mitigating the training gap. 


University of California, Berkeley

Teacher-student architecture is a carrier to form the knowledge transfer. Focused on the relationship between the teacher and student model structure, it consists of selecting and designing proper structures of teacher and student. 


Teacher-Student Architecture


Teacher and student model setup are almost pre-fixed in size and structure during distillation, causing a model capacity gap. It is missing a way to design the architectures and an explanation for why their architectures are determined by these model setups. 


Learn Before

Related

Learn After