Model transfer of knowledge can be extended to other tasks, e.g. adversarial attacks, data augmentation, data privacy and security. The idea of knowledge transfer can be applied to dataset distillation, which transfers the knowledge from a large dataset to a smaller one. 


University of California, Berkeley

Extensions of knowledge distillation focus on compressing deep neural networks. 


Extensions

Gou, J., Yu, B., Maybank, S.J., & Tao, D. (2021). Knowledge Distillation: A Survey. Int. J. Comput. Vis., 129, 1789-1819.

Link: https://arxiv.org/pdf/2006.05525.pdf

Knowledge Distillation: A Survey

Recent methods include teacher-student learning, assistant teaching, lifelong learning, and self-learning. 


Learn Before

Related