Recent methods include teacher-student learning, assistant teaching, lifelong learning, and self-learning. 


University of California, Berkeley

Extensions of knowledge distillation focus on compressing deep neural networks. 


Extensions

Gou, J., Yu, B., Maybank, S.J., & Tao, D. (2021). Knowledge Distillation: A Survey. Int. J. Comput. Vis., 129, 1789-1819.

Link: https://arxiv.org/pdf/2006.05525.pdf

Knowledge Distillation: A Survey

Extension of Methods 


Model transfer of knowledge can be extended to other tasks, e.g. adversarial attacks, data augmentation, data privacy and security. The idea of knowledge transfer can be applied to dataset distillation, which transfers the knowledge from a large dataset to a smaller one. 


Learn Before

Related