- Object detection: Example response: logits with a bounding box offset 
- Semantic landmark localization: Example teacher response: a heat map for each landmark
- Soft targets are the class probabilities of the input. It is the most popular response-based knowledge for image classification.

Example Use Cases 


The distillation loss for response-based knowledge transfer is calculated as the divergence between the logit vectors from the teacher and student models. This can be formally expressed as $$L_{ResD}(z_t, z_s) = L_R(z_t, z_s)$$, where $$z_t$$ and $$z_s$$ are the logits from the teacher and student models, respectively, and $$L_R(\cdot)$$ represents the divergence loss function.

Distillation Loss for Response-Based Knowledge

The benchmark model in knowledge distillation employs a joint loss function that combines the distillation loss and the student loss. The student loss is typically the cross-entropy loss between the ground truth label and the soft logits of the student model, expressed as $$L_{CE}(y, p(z_s, T = 1))$$.

Benchmark Model in Knowledge Distillation

Response-based knowledge distillation relies exclusively on the last layer’s output, meaning it does not encapsulate intermediate-level supervision from the teacher model. Furthermore, this approach is generally limited to supervised learning contexts because the soft logits it depends upon represent class probability distributions.

Drawbacks of Response-Based Knowledge

Response-based knowledge in knowledge distillation consists of a teacher model's final-layer outputs—such as logits, class probabilities, bounding-box offsets, or landmark heat maps—that guide a student model. The student is trained to reduce a divergence between its responses and the teacher's; it does not necessarily copy the teacher's final prediction exactly.

University of California, Berkeley

Google

There are different types of knowledge in knowledge distillation (KD). They are 
- Response based knowledge
- Feature-based knowledge
- Relation-based knowledge  


Categories of Knowledge 

Feature-based knowledge is a knowledge distillation approach that uses the feature maps produced by a teacher model's intermediate layers, not just its final output, as knowledge to transfer to a student model. It builds on the concept of representation learning, since deep networks learn multiple levels of increasingly abstracted feature representations, and it is especially suitable for training thinner and deeper student networks.

Feature-based knowledge

Response-Based Knowledge in Knowledge Distillation

Relation-based knowledge in knowledge distillation consists of relationships represented by a teacher model—such as pairwise similarities, feature-map correlations, probabilistic relations among samples, or instance relationship graphs—that a student model learns to preserve. It focuses on relations among representations or samples rather than isolated layer outputs.

Learn Before

Related

Learn After