Computational Infeasibility of Full Output Summation in Distillation Loss

A student model is trained to mimic a teacher model by minimizing the following loss function, which measures the dissimilarity between their output probability distributions for a given input:

$\text{Loss} = -\sum_{\mathbf{y}} \text{Pr}^t(\mathbf{y}) \log \text{Pr}_{\theta}^s(\mathbf{y})$

In this formula, $\text{Pr}^t(\mathbf{y})$ is the teacher's probability for an output sequence $\mathbf{y}$, $\text{Pr}_{\theta}^s(\mathbf{y})$ is the student's probability, and the summation is over all

Evaluating a Loss Function for a Machine Translation Task

A student model is being trained to replicate the output distribution of a teacher model using the loss function:

$\text{Loss} = -\sum_{\mathbf{y}} \text{Pr}^t(\mathbf{y}) \log \text{Pr}_{\theta}^s(\mathbf{y})$

Suppose for a given input, there are only three possible output sequences: A, B, and C. The teacher model assigns the following probabilities:

• Pr^t(A) = 0.8
• Pr^t(B) = 0.15
• Pr^t(C) = 0.05

Two different student models produce the following distributions:

• Student 1: