- Depth-Wise Separable convolution are used to design efficient neural networks for mobile or embedded devices. 
- Improving the performance of small neural networks by searching for a global structure based on meta operations or block. 
- Removing redundant layers using reinforcement in KD and searching for optimal student networks using the teacher networks. 
- Neural architecture search in KD is a teacher-guided, joint search of student structure and knowledge transfer 

Examples of Recent Work


Recent knowledge-distillation work considers the teacher architecture, student architecture, and knowledge-transfer scheme as connected design choices. An adaptive teacher-student learning architecture can coordinate these choices to better match the teacher and student models.

University of California, Berkeley

Google

- Add a teacher assistant mitigates the training gap, which is further reduced by residual learning, where the assistant structure is used to learn the residual error. 
- Minimize the difference in teacher and student model structure by combining network quantization with knowledge distillation.
- Structure compression method: Transfers the knowledge of multiple layers to a single layer. In online settings, teacher networks are ensembles of similarly-structured student networks. 

Techniques for Mitigating the Training Gap 


In traditional knowledge distillation, the teacher and student model architectures are almost always pre-fixed in size and structure. This fixed configuration often results in a model capacity gap, where the smaller student cannot effectively absorb the larger teacher's knowledge. Furthermore, standard distillation frameworks lack a systematic methodology to design these architectures and do not explain how the pre-fixed model setups constrain the architectural choices.

Learn Before

Related

Learn After