- Depth-Wise Separable convolution are used to design efficient neural networks for mobile or embedded devices. 
- Improving the performance of small neural networks by searching for a global structure based on meta operations or block. 
- Removing redundant layers using reinforcement in KD and searching for optimal student networks using the teacher networks. 
- Neural architecture search in KD is a teacher-guided, joint search of student structure and knowledge transfer 

University of California, Berkeley

Recent work mostly focus on designing either the teacher or student model structures or the knowledge transfer scheme. To match the two models, the adaptive teacher-student learning architecture is necessary. 


Learn Before

Related