The benchmark model in knowledge distillation employs a joint loss function that combines the distillation loss and the student loss. The student loss is typically the cross-entropy loss between the ground truth label and the soft logits of the student model, expressed as $$L_{CE}(y, p(z_s, T = 1))$$.

University of California, Berkeley

Google

Response based knowledge is simple, effective, and widely-applicable. It refers to the neural response of the teacher model’s last output layer. It copies the teacher model’s final prediction exactly. 


Response based knowledge


- Object detection: Example response: logits with a bounding box offset 
- Semantic landmark localization: Example teacher response: a heat map for each landmark
- Soft targets are the class probabilities of the input. It is the most popular response-based knowledge for image classification.

Example Use Cases 



Response-based knowledge relies on the last layer’s output, so it does not encapsulate the intermediate level supervision from the teacher model. The response based knowledge distillation is limited to supervised learning because soft logits are the class probability distribution. 


Drawbacks

The distillation loss for response-based knowledge transfer is calculated as the divergence between the logit vectors from the teacher and student models. This can be formally expressed as $L_{ResD}(z_t, z_s) = L_R(z_t, z_s)$, where $z_t$ and $z_s$ are the logits from the teacher and student models, respectively, and $L_R(.)$ represents the divergence loss function.

Learn Before

Related