Teacher and student model setup are almost pre-fixed in size and structure during distillation, causing a model capacity gap. It is missing a way to design the architectures and an explanation for why their architectures are determined by these model setups. 


University of California, Berkeley

Teacher-student architecture is a carrier to form the knowledge transfer. Focused on the relationship between the teacher and student model structure, it consists of selecting and designing proper structures of teacher and student. 


Teacher-Student Architecture


Recent work 


Model capacity gap between the two models can degrade knowledge transfer. Methods for controlling the reduction of model complexity include techniques for mitigating the training gap. 


Learn Before

Related