In a knowledge distillation framework, a larger and more powerful 'teacher' model is used to train a 'student' model that is designed to be smaller and more efficient. The teacher model processes a full-context user input to generate its output probability, denoted as $Prt(y|c, z)$. In contrast, the student model processes a simplified context input to produce its own output, $Prs(y|c', z)$. The training objective is to transfer knowledge from the stronger teacher to the compact student by minimizing a loss function that measures the difference between their respective outputs.

Teacher-Student Model Architecture in Knowledge Distillation

The key problem is figuring how to transfer the knowledge from a teacher model to a student model.


University of California, Berkeley

The teacher learns relationships between activations, neurons, or sample pairs. The logits or parameters of a large deep model are used as the teacher knowledge. Activations, neurons, or features of intermediate layers can also guide learning. 



KD Workflow 

Gou, J., Yu, B., Maybank, S.J., & Tao, D. (2021). Knowledge Distillation: A Survey. Int. J. Comput. Vis., 129, 1789-1819.

Link: https://arxiv.org/pdf/2006.05525.pdf

Learn Before

Related

Learn After