Recent work mostly focus on designing either the teacher or student model structures or the knowledge transfer scheme. To match the two models, the adaptive teacher-student learning architecture is necessary. 


Recent Work 


- Add a teacher assistant mitigates the training gap, which is further reduced by residual learning, where the assistant structure is used to learn the residual error. 
- Minimize the difference in teacher and student model structure by combining network quantization with knowledge distillation.
- Structure compression method: Transfers the knowledge of multiple layers to a single layer. In online settings, teacher networks are ensembles of similarly-structured student networks. 

University of California, Berkeley

Model capacity gap between the two models can degrade knowledge transfer. Methods for controlling the reduction of model complexity include techniques for mitigating the training gap. 


Learn Before

Related

Learn After