- Depth-Wise Separable convolution are used to design efficient neural networks for mobile or embedded devices. 
- Improving the performance of small neural networks by searching for a global structure based on meta operations or block. 
- Removing redundant layers using reinforcement in KD and searching for optimal student networks using the teacher networks. 
- Neural architecture search in KD is a teacher-guided, joint search of student structure and knowledge transfer 

Examples of Recent Work


Recent work mostly focus on designing either the teacher or student model structures or the knowledge transfer scheme. To match the two models, the adaptive teacher-student learning architecture is necessary. 


University of California, Berkeley

- Add a teacher assistant mitigates the training gap, which is further reduced by residual learning, where the assistant structure is used to learn the residual error. 
- Minimize the difference in teacher and student model structure by combining network quantization with knowledge distillation.
- Structure compression method: Transfers the knowledge of multiple layers to a single layer. In online settings, teacher networks are ensembles of similarly-structured student networks. 

Techniques for Mitigating the Training Gap 


In traditional knowledge distillation, the teacher and student model architectures are almost always pre-fixed in size and structure. This fixed configuration often results in a model capacity gap, where the smaller student cannot effectively absorb the larger teacher's knowledge. Furthermore, standard distillation frameworks lack a systematic methodology to design these architectures and do not explain how the pre-fixed model setups constrain the architectural choices.

Learn Before

Related

Learn After