An engineer is training a small 'student' model to mimic the predictions of a larger, pre-trained 'teacher' model. The training objective is to make the student's final, pre-activation output vector as similar as possible to the teacher's. If z_t is the teacher's output vector and z_s is the student's output vector for the same input, which of the following loss functions correctly implements this objective?

Rationale for Using Logits in Distillation Loss

When implementing response-based knowledge distillation, the loss function is calculated by first applying a softmax function to the teacher and student model outputs to convert them into probability distributions, and then measuring the divergence between these two distributions.