- Object detection: Example response: logits with a bounding box offset 
- Semantic landmark localization: Example teacher response: a heat map for each landmark
- Soft targets are the class probabilities of the input. It is the most popular response-based knowledge for image classification.

Example Use Cases 


The benchmark model is the joint of the distillation and student losses. Student loss is always the cross entropy loss (below) between the ground truth label and soft logits of the student model. 

$L_{CE}(y, p(z_s, T = 1))$

Benchmark Model 


Response-based knowledge relies on the last layer’s output, so it does not encapsulate the intermediate level supervision from the teacher model. The response based knowledge distillation is limited to supervised learning because soft logits are the class probability distribution. 


Drawbacks

The distillation loss for response-based knowledge transfer is calculated as the divergence between the logit vectors from the teacher and student models. This can be formally expressed as $L_{ResD}(z_t, z_s) = L_R(z_t, z_s)$, where $z_t$ and $z_s$ are the logits from the teacher and student models, respectively, and $L_R(.)$ represents the divergence loss function.

Distillation Loss for Response-Based Knowledge

Response based knowledge is simple, effective, and widely-applicable. It refers to the neural response of the teacher model’s last output layer. It copies the teacher model’s final prediction exactly. 


University of California, Berkeley

There are different types of knowledge in knowledge distillation (KD). They are 
- Response based knowledge
- Feature-based knowledge
- Relation-based knowledge  


Categories of Knowledge 

Response based knowledge


Feature-based knowledge builds on the concept of representation learning. It is especially suitable for training thinner and deeper networks. 

Feature-based knowledge



Relation-based knowledge investigates the relationships between different layers or data samples, rather than outputs of specific layers. 

Learn Before

Related

Learn After