The benchmark model is the joint of the distillation and student losses. Student loss is always the cross entropy loss (below) between the ground truth label and soft logits of the student model. 

$L_{CE}(y, p(z_s, T = 1))$

University of California, Berkeley

Response based knowledge is simple, effective, and widely-applicable. It refers to the neural response of the teacher model’s last output layer. It copies the teacher model’s final prediction exactly. 


Response based knowledge


- Object detection: Example response: logits with a bounding box offset 
- Semantic landmark localization: Example teacher response: a heat map for each landmark
- Soft targets are the class probabilities of the input. It is the most popular response-based knowledge for image classification.

Example Use Cases 


Benchmark Model 


Response-based knowledge relies on the last layer’s output, so it does not encapsulate the intermediate level supervision from the teacher model. The response based knowledge distillation is limited to supervised learning because soft logits are the class probability distribution. 


Drawbacks

The distillation loss for response-based knowledge transfer is calculated as the divergence between the logit vectors from the teacher and student models. This can be formally expressed as $L_{ResD}(z_t, z_s) = L_R(z_t, z_s)$, where $z_t$ and $z_s$ are the logits from the teacher and student models, respectively, and $L_R(.)$ represents the divergence loss function.

Learn Before

Related