In response-based knowledge distillation, the teacher response depends on the task. For image classification, soft targets are class probabilities; for object detection, responses can include class logits and bounding-box offsets; and for semantic landmark localization, the teacher can produce a heat map for each landmark.

University of California, Berkeley

Google

Response-based knowledge in knowledge distillation consists of a teacher model's final-layer outputs—such as logits, class probabilities, bounding-box offsets, or landmark heat maps—that guide a student model. The student is trained to reduce a divergence between its responses and the teacher's; it does not necessarily copy the teacher's final prediction exactly.

Response-Based Knowledge in Knowledge Distillation

The distillation loss for response-based knowledge transfer is calculated as the divergence between the logit vectors from the teacher and student models. This can be formally expressed as $$L_{ResD}(z_t, z_s) = L_R(z_t, z_s)$$, where $$z_t$$ and $$z_s$$ are the logits from the teacher and student models, respectively, and $$L_R(\cdot)$$ represents the divergence loss function.

Distillation Loss for Response-Based Knowledge

The benchmark model in knowledge distillation employs a joint loss function that combines the distillation loss and the student loss. The student loss is typically the cross-entropy loss between the ground truth label and the soft logits of the student model, expressed as $$L_{CE}(y, p(z_s, T = 1))$$.

Benchmark Model in Knowledge Distillation

Response-based knowledge distillation relies exclusively on the last layer’s output, meaning it does not encapsulate intermediate-level supervision from the teacher model. Furthermore, this approach is generally limited to supervised learning contexts because the soft logits it depends upon represent class probability distributions.

Learn Before

Related