Cross-Entropy Loss for Knowledge Distillation

Using KL Divergence for Knowledge Distillation Loss

A research team is training a small, efficient 'student' model to replicate the behavior of a large, powerful 'teacher' model. The team's goal is to find the optimal parameters for the student model ($\hat{\theta}$) by minimizing a loss function over a dataset of simplified inputs ($\mathcal{D}'$), as defined by the following objective:

$\hat{\theta} = \arg\min_{\theta} \sum_{\mathbf{x}' \in \mathcal{D}'} \text{Loss}(\text{Pr}^t(\cdot|\cdot), \text{Pr}_{θ}^s(\cdot|\cdot), \mathbf{x}')$

Where $

Critique of a Modified Training Objective

Diagnosing a Knowledge Distillation Training Issue