- Depth-Wise Separable convolution are used to design efficient neural networks for mobile or embedded devices. 
- Improving the performance of small neural networks by searching for a global structure based on meta operations or block. 
- Removing redundant layers using reinforcement in KD and searching for optimal student networks using the teacher networks. 
- Neural architecture search in KD is a teacher-guided, joint search of student structure and knowledge transfer 

Examples of Recent Work


Recent work mostly focus on designing either the teacher or student model structures or the knowledge transfer scheme. To match the two models, the adaptive teacher-student learning architecture is necessary. 


University of California, Berkeley

- Add a teacher assistant mitigates the training gap, which is further reduced by residual learning, where the assistant structure is used to learn the residual error. 
- Minimize the difference in teacher and student model structure by combining network quantization with knowledge distillation.
- Structure compression method: Transfers the knowledge of multiple layers to a single layer. In online settings, teacher networks are ensembles of similarly-structured student networks. 

Learn Before

Related

Learn After